Backup procedures
From Wikitech
(Difference between revisions)
(→Soft data) |
(→Software development and configuration) |
||
| (8 intermediate revisions by 3 users not shown) | |||
| Line 104: | Line 104: | ||
||[[db10]] replication | ||[[db10]] replication | ||
|| EQIAD [[db1008]]/[[db1025]] & snapshots | || EQIAD [[db1008]]/[[db1025]] & snapshots | ||
| − | || | + | || db10.pmtpa.w /a/sqldata 0 66532680 9020197 13.6 31:49 4724.1 31:49 4724.1 |
|| 1.5 GB | || 1.5 GB | ||
|- | |- | ||
| Line 188: | Line 188: | ||
|| PMTPA various | || PMTPA various | ||
|| EQIAD is primary now | || EQIAD is primary now | ||
| − | || | + | || SDTPA |
|| | || | ||
|- | |- | ||
| Line 194: | Line 194: | ||
|| PMTPA [[locke]] | || PMTPA [[locke]] | ||
|| emery | || emery | ||
| − | |||
|| /a/squid | || /a/squid | ||
| − | || | + | || oxygen (eqiad) |
|- | |- | ||
|| MediaWiki logs | || MediaWiki logs | ||
|| PMTPA [[NFS]] | || PMTPA [[NFS]] | ||
|| [[tridge]] | || [[tridge]] | ||
| − | | | + | || [[fluorine]] |
|| /home/wikipedia/logs | || /home/wikipedia/logs | ||
|| | || | ||
| Line 232: | Line 231: | ||
|- | |- | ||
|| Impression Logs | || Impression Logs | ||
| − | || PMTPA [[ | + | || PMTPA [[nas1]] |
| − | | | + | || |
| − | + | || [[nas1001]] | |
|| | || | ||
|| 1.5TB year | || 1.5TB year | ||
| Line 270: | Line 269: | ||
|| Daily backups to Tridge | || Daily backups to Tridge | ||
|| 30 GB | || 30 GB | ||
| + | |- | ||
| + | || stat1 /a and /home | ||
| + | || - | ||
| + | || [[Amanda]] to [[tridge]] | ||
| + | |class="missing"| | ||
| + | || [https://gerrit.wikimedia.org/r/#/c/9258/3 gerrit change] | ||
| + | || 400 GB | ||
|} | |} | ||
Latest revision as of 21:43, 26 October 2012
This page is an inventory of our backups for both clusters data and office data.
Some definitions:
- On-site
- here means the backup is in the same physical location as the master, but on a different machine.
- Off-site
- means the backup is in a physically separate location. In most cases it's nice to have both!
- Red cells
- denote a lacking area which we want/need.
Contents |
[edit] Service backup state
| Service | Master | On-site | Off-site | Notes | Approx. Space Needed |
|---|---|---|---|---|---|
[edit] Wiki primary data | |||||
| Wiki main databases | PMTPA db* | PMTPA slave replication | EQIAD db slaves & snapshots / ESAMS toolserver replication | http://noc.wikimedia.org/dbtree/ | 4 TB |
| External storage | PMTPA db* | PMTPA slave replication | EQIAD ES1004 replicates ms03; ES1001-3 is a copy of rc1 & cl1-cl21 /ESAMS toolserver replication | 5 TB | |
| Wiki data dumps | PMTPA dataset2 | Gluster storage (labstore1 under /publicdata-project directory) Last 5 good only. | EQIAD: dataset1001 (via rsync). All public data: dumps.wikimedia.your.org. Last 5 good dumps: wikipedia.c3sl.ufpr.br (ftp/http/rsync). | See Dumps/Mirror status for more info. | 6T (last 5 good)/ 29T (all) |
[edit] Images and media | |||||
| Uploads | PMTPA ms7 | PMTPA ms8 | EQIAD ms1002 / ESAMS ms6 (partial) | 23T | |
| Thumbs | PMTPA ms5 | EQIAD ms1004 | 12T | ||
[edit] Software development and configuration | |||||
| MediaWiki config | PMTPA NFS | tridge | Subversion | ||
| Apache config | PMTPA NFS | tridge | Automatic | rsync | |
| Subversion | Formey | tridge | 3 GB | ||
| Gerrit | Manganese | Formey | Databases on db9/10 | 3 GB | |
| Bugzilla data | PMTPA db9 | db10 replication | EQIAD db1008/db1025 & snapshots | 1 GB | |
| Bugzilla config | PMTPA db9 | db10 replication | EQIAD db1008/db1025 & snapshots | db10.pmtpa.w /a/sqldata 0 66532680 9020197 13.6 31:49 4724.1 31:49 4724.1 | 1.5 GB |
| Bugzilla frontend | PMTPA isidore | daily rsync to Tridge | |||
| Wikitech wiki | linode | PMTPA tridge; ESAMS ? | |||
[edit] Communications | |||||
| OTRS data | PMTPA db9 | db10 replication | EQIAD db1008/db1025 & snapshots | 57 GB | |
| OTRS config | PMTPA bart | 19 GB | |||
| blogs frontend data | PMTPA singer | Daily backups to Tridge | |||
| bugzilla | PMTPA isidore | Daily backups to Tridge | |||
| IMAP mail | PMTPA sanger | PMTPA mchenry, Daily backups to tridge | see Mail#Backups | 32 GB | |
| Mailing lists | ESAMS lily - decommissioned | EQIAD sodium | PMTPA mchenry | See Mailing lists#Backups | 50 GB |
| DNS | PMTPA ns0, ns1 | checkin to sockpuppet SVN | ESAMS ns2 | In SVN now | |
| Google docs | Google :P | ad-hoc downloads | |||
[edit] Soft data | |||||
| Search databases | PMTPA various | EQIAD is primary now | SDTPA | ||
| HTTP logs | PMTPA locke | emery | /a/squid | oxygen (eqiad) | |
| MediaWiki logs | PMTPA NFS | tridge | fluorine | /home/wikipedia/logs | |
[edit] Fundraising | |||||
| Fundraising Front-End | EQIAD aluminium | software:svn logs:storage3->tridge |
puppet:files/misc/scripts/offhost_backups | 5GB | |
| Fundraising Databases | EQIAD db1008 | db1025 replication + snapshots | PMTPA storage3 replication, snapshots, dumps tridge daily rsync of dumps |
puppet:files/misc/scripts/dump_fundraisingdb | 1TB (15GB/day) |
| Payment Processing | software:svn db+logs:PMTPA |
db:mysql replication db+logs:encrypt+copy daily to tridge |
db+logs:replicate to silicon | payments cluster | 100GB(?) |
| Impression Logs | PMTPA nas1 | nas1001 | 1.5TB year | ||
[edit] Misc data | |||||
| Office fileserver | OFFICE fileserver | weekly rsync to USB drive | rsync to tridge not running | 2 GB(?) | |
| Office workstations | OFFICE * | currently no backups done (can be enabled, if needed; MacOS only) | |||
| Office laptops | OFFICE * | Time Machine backups to usb drive on Imac (MacOS only) | |||
| Server home directories | PMTPA nfs1 / nfs2 | tridge | Daily backups to Tridge | 30 GB | |
| stat1 /a and /home | - | Amanda to tridge | gerrit change | 400 GB | |
