Dumps/Snapshot hosts
From Wikitech
< Dumps(Difference between revisions)
ArielGlenn (Talk | contribs) |
ArielGlenn (Talk | contribs) (→Currently running) |
||
| Line 26: | Line 26: | ||
===Currently running=== | ===Currently running=== | ||
| − | * | + | Monitors: |
| + | |||
| + | * snapshot1 -- current monitor node: | ||
| + | *:<code>/bin/bash ./monitor wikidump.conf.monitor</code> | ||
| + | |||
| + | Worker nodes: | ||
| + | |||
| + | * snapshot1 -- currently running 3 worker processes for bigger wikis out of /backups-atg, via | ||
*:<code>python ./worker wikidump.conf.bigwikis</code> | *:<code>python ./worker wikidump.conf.bigwikis</code> | ||
| − | * | + | * snapshot2 -- running 4 processes for small wikis out of /backups-atg, via |
*: <code>./worker</code> | *: <code>./worker</code> | ||
| − | * | + | * snapshot3 -- runs adds/changes dumps from cron |
| − | * | + | * snapshot4 -- running en wiki dumps via |
*: <code>python ./worker.py --configfile wikidump.conf.enwiki enwiki</code> | *: <code>python ./worker.py --configfile wikidump.conf.enwiki enwiki</code> | ||
[[Category:Dumps]] | [[Category:Dumps]] | ||
Revision as of 18:41, 19 January 2012
Snapshot (XML dumps generation) cluster information
Hardware
These hosts generate the XML dumps. For information about the hosts that serve them, see Dumps/Dump servers.
We have two mini snapshot clusters.
In Tampa:
- snapshot1: operational, PowerEdge 1950, Ubuntu 10.04, 8GB RAM, 2 quad-core Xeons, 80GB HD
- snapshot2: operational, PowerEdge 1950, Ubuntu 10.04, 8GB RAM, 2 quad-core Xeons, 80GB HD
- snapshot3: operational, PowerEdge 1950, Ubuntu 10.04, 8GB RAM, 2 quad-core Xeons, 80GB HD
- snapshot4: operational, PowerEdge R815, Ubuntu 10.04, 8GB RAM, 4 8-core Opterons, 2 80GB HDs
In D.C.:
- snapshot1001: base install done, PowerEdge R815, Ubuntu 10.04, 64GB RAM, 4 8-core Opterons, 2 80GB HDs
- snapshot1002: base install done, PowerEdge R410, Ubuntu 10.04, 16GB RAM, 2 6-core Xeons, 500GB HD
- snapshot1003: waiting for drac issue fix, PowerEdge R410, Ubuntu 10.04, 16GB RAM, 2 6-core Xeons, 500GB HD
- snapshot1004: waiting for drac issue fix, PowerEdge R410, Ubuntu 10.04, 16GB RAM, 2 6-core Xeons, 500GB HD
Ordinarily only one cluster will be running dump jobs at a time; the other is on standby in case of various failures. The two beefier servers (with 4 8-core cpus) are dedicated machines for the en wikipedia dumps; as with the other hosts, one of them is in opertion and the other is in standby.
Currently running
Monitors:
- snapshot1 -- current monitor node:
/bin/bash ./monitor wikidump.conf.monitor
Worker nodes:
- snapshot1 -- currently running 3 worker processes for bigger wikis out of /backups-atg, via
python ./worker wikidump.conf.bigwikis
- snapshot2 -- running 4 processes for small wikis out of /backups-atg, via
-
./worker
-
- snapshot3 -- runs adds/changes dumps from cron
- snapshot4 -- running en wiki dumps via
-
python ./worker.py --configfile wikidump.conf.enwiki enwiki
-
