Dumps

Some quick notes on new/old/broken/experimental stuff as of 2005-09-03

Top-level procedure

On benet as root in a screen session, run:

 # run-backup | tee some-file-if-you-like

This will iterate through the backup-XXX scripts for special, wikipedia, etc. It may or may not run all the proper wikis.

Each individual wiki gets backed up by the backup-wiki shell script (currently an alias to backup-wiki-1.6). I've recently revamped this script to be, I hope, more readable and maintainable.

It probably still has flaws.

Locks and logs

While each backup runs it should have a backup.lock file open in the dump directory (eg /var/backup/public/special/meta). It will also write status lines to backup.log.

On error conditions it will write an ABORT line to the log and abort that individual wiki's backup. Currently other backups will continue (and may all fail if there's a general problem). In theory on abort it should also e-mail me but that seems broken at the moment; the dest address can be set in backup-wiki-1.6 in the config at the top.

If the lock file is there, a second run will skip over that wiki; you could use this to ignore particular wikis for instance. An abort should leave the lock file there.

Programs used

• A bunch of mysqldump's for various private and public tables
• dumpBackup.php to generate the XML dump
  • Requires working PHP and MediaWiki installation on benet! Don't remove from mediawiki_install dsh group!
• mwdumper to filter and compress the dump into pages_current.xml.gz, pages_public.xml.gz, and pages_full.xml.gz
  • Requires mono (installed from rpms, there are some in /h/w/src)
  • mwdumper source in CVS, makefile installs into /usr/local
• tar for upload dump
  • Requires /mnt/wikipedia to be mounted... and maybe /mnt/upload for en?
• md5sum
  • wooooo

Notes

Not all error detection is probably working right now. Failures on the mysqldump runs are not detected. Tar failures are not detected.

Failures of dumpPages.php should be detected, but indirectly from the failure of mwdumper to parse its XML output.

The mysql dumps and page XML dump pull from bacon, as configured in one of the higher-level backup scripts. Currently bacon is *NOT* being stopped from replication...

• The page XML dumps should be consistent, all three outputs draw from one input, which is drawn from one long SQL transaction plus supplementary data loads which should be independent of changes.
• The other SQL dumps are not going to be 100% time-consistent. But that's not too important.

grantswiki and internalwiki are special-cased so they _should_ get completely backed up into /var/backup/private instead of the public dir.