Dumps/Adds-changes dumps
Contents |
Adds/changes dumps overview
We have an experimental service available which produces dumps of added/changed content on a daily basis for all projects that have not been closed and are not private.
The code for this service is available in subversion.
The job runs in three phases, currently on snapshot3, as the backup user. These three phases are run from cron, from a copy of the scripts that live in the directory /backups-atg/incrementals on that host.
Directory structure:
Everything for a given run is stored in dumproot/projectname/yyyymmdd/ much as we do for regular dumps.
Phase one
This job runs around 7 am UTC time. We record the largest revision id for the given project, in the file maxrevid.txt.
Phase two
This job runs around 7 pm UTC time, a delay of 12 hours. At this writing it takes about 90 minutes or so to complete. We generate a stubs file containing metadata in xml format for each revision added since the previous day, consulting the file maxrevid.txt for the previous day to get the start of the range. We then generate a meta-history xml file which contains the text of these revisions grouped together and sorted by page id. Md5 sums of these are available in an md5sums.txt file. A status.txt file is available to indicate whether we had a successful run ("done") or not.
Phase three
This job runs around 11:30 pm UTC, leaving hopefully plenty of time for the previous job to complete. It checks the directories for successful runs and writes a main index.html file with links for each project to the stub and content files for the latest successful run.
When stuff breaks
You can rerun various jobs by hand for specified dates. More info coming soon...
Some numbers
Here's a few fun numbers from the November 23 2011 run. Writing the stubs file for 167985 revisions for en wikipedia took 2 minutes, and writing the revisions text file took 24 minutes. Writing the stubs file for 36272 revisions for de wikipedia took less than a minute, and writing the revisions text file took 5 minutes. Writing the stubs file for 43133 revisions for commons took 1 minute, and writing the revisions text file took 2 minutes.
