View source for Dumps/Adds-changes dumps

==Adds/changes dumps overview==

We have an <strong>experimental</strong> service available which produces dumps of added/changed content on a daily basis for all projects that have not been closed and are not private.

The code for this service is available in [http://svn.wikimedia.org/viewvc/mediawiki/branches/ariel/xmldumps-backup/incrementals/ subversion].  

The job runs in three phases, currently on snapshot3, as the ''backup'' user.  These three phases are run from cron, from a copy of the scripts that live in the directory /backups-atg/incrementals on that host.

Directory structure:

Everything for a given run is stored in dumproot/projectname/yyyymmdd/ much as we do for regular dumps.

===Phase one===

This job runs around 7 am UTC time.  We record the largest revision id for the given project, in the file maxrevid.txt.

===Phase two===

This job runs around 7 pm UTC time, a delay of 12 hours. At this writing it takes about 90 minutes or so to complete.  We generate a stubs file containing metadata in xml format for each revision added since the previous day, consulting the file maxrevid.txt for the previous day to get the start of the range.  We then generate a meta-history xml file which contains the text of these revisions grouped together and sorted by page id.  Md5 sums of these are available in an md5sums.txt file.  A status.txt file is available to indicate whether we had a successful run ("done") or not.

===Phase three===

This job runs around 11:30 pm UTC, leaving hopefully plenty of time for the previous job to complete.  It checks the directories for successful runs and writes a main index.html file with links for each project to the stub and content files for the latest successful run.

==When stuff breaks==

You can rerun various jobs by hand for specified dates.  

From the directory /backups-atg/incrementals, as the backup user, you can run 

*<code>python ./generatemaxrevids.py</code> to retrieve the maxrevids for ''today'' at the time of the run. (avoid this when possible, let cron do it at the scheduled hour.)
*<code>python ./generateincrementals.py yyyymmdd</code> to generate the stubs and revs text files for a given date; this presumes that the revids form the previous step are already recorded in the file maxrevids.txt in the directory for the given date and project.  You can add --verbose to get information about what it's doing.  If it complains about lock files in place you can remove these by hand, providing that the cron job is not running at the time and there is no other copy of this job running.
*<code>python ./incrmonitor.py</code> to regenerate the index.html file listing all projects, after the previous two steps are complete. 

===Internals===

Locks:

During phase one, the retrieval of the max revision id from each project in turn, the job attempts to create a lock file with name in the format dbname-yyyymmdd-maxrevid.lock in the root of the directory tree for that project's add/changes dumps.  For example, a job running on Nov. 24 2011 for frwiki would try to create the lockfile frwiki/20111124/frwiki-20111124-maxrevid.lock.

During phase two, the creation of stub and content xml files, the job tries to create a lockfile in the top level directory for the add/changes dumps for the given project, with name in the format dbname-yyyymmdd-incrdump.lock.

===TODOs===

Add stale lock removal option.   Need to write interpolation of maxrevids in case step one fails on some days.  Need to walk through older directories and write older stub/revision files if we see those files don't exist (this allows recovery if the job is out of action or fails for a few days).

Be able to restart (a given date) from a particular wiki, allow option to force removal of locks for the run.

==Some numbers==

Here's a few fun numbers from the November 23 2011 run.  Writing the stubs file for 167985 revisions for en wikipedia took 2 minutes, and writing the revisions text file took 24 minutes.  Writing the stubs file for 36272 revisions for de wikipedia took less than a minute, and writing the revisions text file took 5 minutes.  Writing the stubs file for 43133 revisions for commons took 1 minute, and writing the revisions text file took 2 minutes.

[[Category:Dumps]]