Dumps
From Wikitech
(Difference between revisions)
ArielGlenn (Talk | contribs) (→Worker nodes) |
ArielGlenn (Talk | contribs) |
||
| (101 intermediate revisions by 2 users not shown) | |||
| Line 1: | Line 1: | ||
Docs for end-users of the data dumps at [[MetaWikipedia:Data dumps]]. | Docs for end-users of the data dumps at [[MetaWikipedia:Data dumps]]. | ||
| − | For | + | For a list of various information sources about the dumps, see [[Dumps/Other information sources]]. |
| − | + | *For documentation on the "adds/changes" dumps, see [[Dumps/Adds-changes dumps]]. | |
| + | *For documentation on the media dumps, see [[Dumps/media]]. | ||
| + | *For current development plans, see [[Dumps/Development 2012]]. | ||
| + | *For historical information about the dumps, see [[Dumps/History]]. | ||
| − | + | {| cellspacing="0" cellpadding="0" style="clear: {{{clear|right}}}; margin-bottom: .5em; float: right; padding: .5em 0 .8em 1.4em; background: none; width: {{{width|{{{1|auto}}}}}};" | |
| + | | __TOC__ | ||
| + | |} | ||
| + | == Overview == | ||
User-visible files appear at http://download.wikipedia.org/backup-index.html | User-visible files appear at http://download.wikipedia.org/backup-index.html | ||
| Line 13: | Line 19: | ||
=== Status === | === Status === | ||
| − | + | For which hosts are serving data, see [[Dumps/Dump servers]]. For which hosts are generating which dumps, see [[Dumps/Snapshot hosts]]. | |
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | We want mirrors! For more information see [[Dumps/Mirror status]]. | |
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
=== Worker nodes === | === Worker nodes === | ||
| Line 32: | Line 27: | ||
The worker processes go through the set of available wikis to dump automatically. Dumps are run on a "longest without a dump runs next" schedule. The plan is to have a complete dump for each wiki every 2 weeks, except for enwikipedia, which should have a complete dump once a month. | The worker processes go through the set of available wikis to dump automatically. Dumps are run on a "longest without a dump runs next" schedule. The plan is to have a complete dump for each wiki every 2 weeks, except for enwikipedia, which should have a complete dump once a month. | ||
| − | The shell script <code>worker</code> which starts one of these processes simply runs the python script <worker.py> in an endless loop. | + | The shell script <code>worker</code> which starts one of these processes simply runs the python script <worker.py> in an endless loop. Multiple such workers can run at the same time on different hosts, as well as on the same host. |
The <code>worker.py</code> script creates a lock file on the filesystem containing the dumps (as of this writing, <code>/mnt/data/xmldatadumps/</code>) in the subdirectory <code>private/name-of-wiki/lock</code>. No other process will try to write dumps for that project while the lock file is in place. | The <code>worker.py</code> script creates a lock file on the filesystem containing the dumps (as of this writing, <code>/mnt/data/xmldatadumps/</code>) in the subdirectory <code>private/name-of-wiki/lock</code>. No other process will try to write dumps for that project while the lock file is in place. | ||
| − | Local copies of the shell script and the python script live on the snapshot hosts in the directory <code>/backups</code> | + | Local copies of the shell script and the python script <small>live on the snapshot hosts in the directory <code>/backups</code> but</small> currently are run out of /backups-atg (since this code is not yet in trunk) in screen sessions on the various hosts, as the user "backup". |
| − | + | === Monitor node === | |
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | The monitor node checks for and removes stale lock files from dump processes that have died, and updates the central <code>index.html</code> file which shows the dumps in progress and the status of the dumps that have completed (i.e. <code>http://dumps.wikimedia.org/backup-index.html</code> ). ''It does not start or stop worker processes.'' | |
| − | + | The shell script <code>monitor</code> which starts the process simply runs the python script <code>monitor.py</code> in an endless loop. | |
| − | + | As with the worker nodes, local copies of the shell script and the python script <small>live on the snapshot hosts in the directory <code>/backups</code> but</small> currently are run out of /backups-atg (since this code is not yet in trunk) in a screen session on one host, as the user "backup". | |
| − | + | === Code === | |
| − | = | + | Check [https://gerrit.wikimedia.org/r/gitweb?p=operations/dumps.git;a=tree;f=xmldumps-backup;hb=ariel /operations/dumps.git, branch 'ariel'] for the python code in use. Eventually this will make its way back into master; it's still a bit gross right now. |
| − | + | Getting a copy: | |
| + | : <code>git clone https://gerrit.wikimedia.org/r/p/operations/dumps.git</code> | ||
| + | : <code>git checkout ariel</code> | ||
| − | + | Getting a copy as a committer: | |
| − | + | : <code>git clone ssh://<user>@gerrit.wikimedia.org:29418/operations/dumps.git</code> | |
| − | + | : <code>git checkout ariel</code> | |
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | == | + | === Programs used === |
| − | + | See also [[Dumps/Software dependencies]]. | |
| − | + | The scripts call mysqldump, getSlaveServer.php, eval.php, dumpBackup.php, and dumpTextPass.php directly for dump generation. These in turn require backup.inc and backupPrefetch.inc and may call ActiveAbstract/AbstractFilter.php and fetchText.php. | |
| − | + | The generation of XML files relies on Export.php under the hood and of course the entire MW infrastructure. | |
| − | + | The worker.py script relies on a few C programs for various bz2 operations: checkforbz2footer and recompressxml, both in /usr/local/bin/. These are in the git repo, see [https://gerrit.wikimedia.org/r/gitweb?p=operations/dumps.git;a=tree;f=xmldumps-backup/mwbzutils;h=e76ee6cb52fd40e570e2e62a969f8b57902de1b9;hb=ariel]. | |
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | == Setup == | |
| − | == | + | === Adding a new worker box === |
| − | + | Install and add to site.pp, copying one of the existing snapshot stanzas in puppet. This does, among other things: | |
| + | # set up the base MW install without apache running | ||
| + | # Add worker to '''/etc/exports/''' on [[dataset2]] | ||
| + | # Add '''/mnt/data''' to '''/etc/fstab''' of worker host | ||
| + | # Build the utfnormal php module (done for lucid) | ||
| − | + | For now: | |
| + | # Backups are running test code out of /backups-atg on each host so grab a copy of that from any existing host and copy it into /backups-atg on the new host. This will include conf files, you don't need to specify them separately. | ||
| + | #: '''In transition, being moved to /backups. To be updated as soon as move is complete.''' | ||
| + | # Check over the configuration file and make sure it looks sane, all the paths point to things that exist, etc. For too many details see [https://gerrit.wikimedia.org/r/gitweb?p=operations/dumps.git;a=blob_plain;f=xmldumps-backup/README.config;hb=ariel the README.config file in the git repo]. | ||
| + | #* We run enwiki on its own host. If this host is going to do that work, check <code>/backups-atg/wikidump.conf.enwiki</code>. | ||
| + | #* The next 8 or so largest wikis are run on their own separate host so they don't backlog the smaller wikis. For that, check <code>/backups-atg/wikidump.conf.bigwikis</code>. | ||
| + | #* The remainder of the wikis run on one host. Check <code>/backups-atg/wikidump.conf</code> for those. | ||
| + | <!--We will eventually do... | ||
| + | # '''git pul something for public repo ... /backups''' | ||
| + | # '''git pull something else for private repo with config files in it... /backups/conf' | ||
| + | # mv wikidump.conf ../.--> | ||
| − | + | == Dealing with problems == | |
| − | == | + | ===Space === |
| + | If the host serving the dumps runs low on disk space, you can reduce the number of backups that are kept. Edit the appropriate file /backups-atg/wikidump.conf* on the host running the set of dumps you would like to adjust, en wiki = wikidump.con.enwiki, the next 8 or so big wikis = wikidump.conf.bigwiki, the rest = wikidump.conf) and change the line that says "keep=<some value>" to some smaller number. | ||
| − | + | ===Failed runs=== | |
| + | Logs will be kept of each run. You can find them in the directory for the particular dump, filename <code>dumplog.txt</code>. You can look at them to see if there are any error messages that were generated for a given run. | ||
| − | + | The worker script can send email if a dump does not complete successfully. (Better enable this.) It currently sends email to... | |
| − | + | When one or more steps of a dump fail, the index.html file for that dump includes a notation of the failure and sometimes more information about it. Note that one step of a dump failing does not prevent other steps from running unless they depend on the data from that failed step as input. | |
| − | + | ||
| − | + | ||
| − | + | See [[Dumps/Rerunning a job]] for how to rerun all or part of a given dump. This also explains what files may need to be cleaned up before rerunning. | |
| − | + | ||
| − | + | ===Dumps not running=== | |
| + | This covers restarting after: rebooting a host, rebooting the dataset host with the nfs share where dumps are written (which may cause dumps to hang), or when the dumps stop running for other reasons. | ||
| − | + | If the host crashes while the script running, the status files are left as-is and the display shows it as still running until the monitor node decides the lock file is stale enough to mark is as aborted. To restart, start a screen session on the host as root and fire up the appropriate number of worker scripts with the appropriate config file option. See [[Dumps/Snapshot hosts]] for which hosts do what; this lists which commands gets run on each host in how many windows. If the monitor script is not running, restart it in a separate window of the same screen session; see the Dump servers page for the command and for which host it runs on. | |
| − | + | ||
| − | + | If the worker script encounters more than three failed dumps in a row (currently configured as such? or did I hardcode that?) it will exit; this avoids generation of piles of broken dumps which later would need to be cleaned up. Once the underlying problem is fixed, you can go to the screen session of the host running those wikis and rerun the previous command in all the windows. See [[Dumps/Snapshot hosts]] for which hosts do what if you're not sure. | |
| − | + | ===Running a specifc dump on request=== | |
| + | See [[Dumps/Rerunning a job]] for how to run a specific dump. This is done for special cases only. | ||
| − | + | == Deploying new code == | |
| − | + | See [[Dumps/How to deploy]] for this. | |
| − | + | == Bugs, known limitations, etc. == | |
| + | See [[Dumps/Known issues and wish list]] for this. | ||
| − | + | == File layout == | |
| − | * | + | * <base>/ |
| − | * / | + | ** [http://dumps.wikimedia.org/index.html index.html] - Information about the server |
| + | ** [http://dumps.wikimedia.org/backup-index.html backup-index.html] - List of all databases and their last-touched status | ||
| + | ** [http://dumps.wikimedia.org/afwiki/ <db>/] | ||
| + | *** <date>/ | ||
| + | **** [http://dumps.wikimedia.org/afwiki/20060122/ index.html] - List of items in the database | ||
| − | + | Sites are identified by raw database name currently. A 'friendly' name/hostname can be added for convenience of searching in future. | |
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | |||
[[Category:How-To]] | [[Category:How-To]] | ||
[[Category:Risk management]] | [[Category:Risk management]] | ||
[[Category:dumps]] | [[Category:dumps]] | ||
Latest revision as of 10:39, 20 June 2012
Docs for end-users of the data dumps at MetaWikipedia:Data dumps.
For a list of various information sources about the dumps, see Dumps/Other information sources.
- For documentation on the "adds/changes" dumps, see Dumps/Adds-changes dumps.
- For documentation on the media dumps, see Dumps/media.
- For current development plans, see Dumps/Development 2012.
- For historical information about the dumps, see Dumps/History.
|
[edit] Overview
User-visible files appear at http://download.wikipedia.org/backup-index.html
Dump activity involves a monitor node (running status sweeps) and arbitrarily many worker nodes running the dumps.
[edit] Status
For which hosts are serving data, see Dumps/Dump servers. For which hosts are generating which dumps, see Dumps/Snapshot hosts.
We want mirrors! For more information see Dumps/Mirror status.
[edit] Worker nodes
The worker processes go through the set of available wikis to dump automatically. Dumps are run on a "longest without a dump runs next" schedule. The plan is to have a complete dump for each wiki every 2 weeks, except for enwikipedia, which should have a complete dump once a month.
The shell script worker which starts one of these processes simply runs the python script <worker.py> in an endless loop. Multiple such workers can run at the same time on different hosts, as well as on the same host.
The worker.py script creates a lock file on the filesystem containing the dumps (as of this writing, /mnt/data/xmldatadumps/) in the subdirectory private/name-of-wiki/lock. No other process will try to write dumps for that project while the lock file is in place.
Local copies of the shell script and the python script live on the snapshot hosts in the directory /backups but currently are run out of /backups-atg (since this code is not yet in trunk) in screen sessions on the various hosts, as the user "backup".
[edit] Monitor node
The monitor node checks for and removes stale lock files from dump processes that have died, and updates the central index.html file which shows the dumps in progress and the status of the dumps that have completed (i.e. http://dumps.wikimedia.org/backup-index.html ). It does not start or stop worker processes.
The shell script monitor which starts the process simply runs the python script monitor.py in an endless loop.
As with the worker nodes, local copies of the shell script and the python script live on the snapshot hosts in the directory /backups but currently are run out of /backups-atg (since this code is not yet in trunk) in a screen session on one host, as the user "backup".
[edit] Code
Check /operations/dumps.git, branch 'ariel' for the python code in use. Eventually this will make its way back into master; it's still a bit gross right now.
Getting a copy:
-
git clone https://gerrit.wikimedia.org/r/p/operations/dumps.git -
git checkout ariel
Getting a copy as a committer:
-
git clone ssh://<user>@gerrit.wikimedia.org:29418/operations/dumps.git -
git checkout ariel
[edit] Programs used
See also Dumps/Software dependencies.
The scripts call mysqldump, getSlaveServer.php, eval.php, dumpBackup.php, and dumpTextPass.php directly for dump generation. These in turn require backup.inc and backupPrefetch.inc and may call ActiveAbstract/AbstractFilter.php and fetchText.php.
The generation of XML files relies on Export.php under the hood and of course the entire MW infrastructure.
The worker.py script relies on a few C programs for various bz2 operations: checkforbz2footer and recompressxml, both in /usr/local/bin/. These are in the git repo, see [1].
[edit] Setup
[edit] Adding a new worker box
Install and add to site.pp, copying one of the existing snapshot stanzas in puppet. This does, among other things:
- set up the base MW install without apache running
- Add worker to /etc/exports/ on dataset2
- Add /mnt/data to /etc/fstab of worker host
- Build the utfnormal php module (done for lucid)
For now:
- Backups are running test code out of /backups-atg on each host so grab a copy of that from any existing host and copy it into /backups-atg on the new host. This will include conf files, you don't need to specify them separately.
- In transition, being moved to /backups. To be updated as soon as move is complete.
- Check over the configuration file and make sure it looks sane, all the paths point to things that exist, etc. For too many details see the README.config file in the git repo.
- We run enwiki on its own host. If this host is going to do that work, check
/backups-atg/wikidump.conf.enwiki. - The next 8 or so largest wikis are run on their own separate host so they don't backlog the smaller wikis. For that, check
/backups-atg/wikidump.conf.bigwikis. - The remainder of the wikis run on one host. Check
/backups-atg/wikidump.conffor those.
- We run enwiki on its own host. If this host is going to do that work, check
[edit] Dealing with problems
[edit] Space
If the host serving the dumps runs low on disk space, you can reduce the number of backups that are kept. Edit the appropriate file /backups-atg/wikidump.conf* on the host running the set of dumps you would like to adjust, en wiki = wikidump.con.enwiki, the next 8 or so big wikis = wikidump.conf.bigwiki, the rest = wikidump.conf) and change the line that says "keep=<some value>" to some smaller number.
[edit] Failed runs
Logs will be kept of each run. You can find them in the directory for the particular dump, filename dumplog.txt. You can look at them to see if there are any error messages that were generated for a given run.
The worker script can send email if a dump does not complete successfully. (Better enable this.) It currently sends email to...
When one or more steps of a dump fail, the index.html file for that dump includes a notation of the failure and sometimes more information about it. Note that one step of a dump failing does not prevent other steps from running unless they depend on the data from that failed step as input.
See Dumps/Rerunning a job for how to rerun all or part of a given dump. This also explains what files may need to be cleaned up before rerunning.
[edit] Dumps not running
This covers restarting after: rebooting a host, rebooting the dataset host with the nfs share where dumps are written (which may cause dumps to hang), or when the dumps stop running for other reasons.
If the host crashes while the script running, the status files are left as-is and the display shows it as still running until the monitor node decides the lock file is stale enough to mark is as aborted. To restart, start a screen session on the host as root and fire up the appropriate number of worker scripts with the appropriate config file option. See Dumps/Snapshot hosts for which hosts do what; this lists which commands gets run on each host in how many windows. If the monitor script is not running, restart it in a separate window of the same screen session; see the Dump servers page for the command and for which host it runs on.
If the worker script encounters more than three failed dumps in a row (currently configured as such? or did I hardcode that?) it will exit; this avoids generation of piles of broken dumps which later would need to be cleaned up. Once the underlying problem is fixed, you can go to the screen session of the host running those wikis and rerun the previous command in all the windows. See Dumps/Snapshot hosts for which hosts do what if you're not sure.
[edit] Running a specifc dump on request
See Dumps/Rerunning a job for how to run a specific dump. This is done for special cases only.
[edit] Deploying new code
See Dumps/How to deploy for this.
[edit] Bugs, known limitations, etc.
See Dumps/Known issues and wish list for this.
[edit] File layout
- <base>/
- index.html - Information about the server
- backup-index.html - List of all databases and their last-touched status
- <db>/
- <date>/
- index.html - List of items in the database
- <date>/
Sites are identified by raw database name currently. A 'friendly' name/hostname can be added for convenience of searching in future.
