Backup procedures

From Wikitech
(Difference between revisions)
Jump to: navigation, search
(→‎Misc data: grosley)
(→‎Software development and configuration)
 
(80 intermediate revisions by 17 users not shown)
Line 1: Line 1:
−
"On-site" here means the backup is in the same physical location as the master, but on a different machine. "Off-site" means the backup is in a physically separate location. In most cases it's nice to have both!
+
This page is an inventory of our backups for both clusters data and office data.
 +
 
 +
Some definitions:
 +
 
 +
; On-site: here means the backup is in the same physical location as the master, but on a different machine.
 +
; Off-site: means the backup is in a physically separate location. In most cases it's nice to have both!
 +
; Red cells: denote a lacking area which we want/need.
 +
 
 +
__TOC__
  
 
== Service backup state ==
 
== Service backup state ==
  
−
{| border="1" class="backup-chart"
+
{| border="1" class="wikitable backup-chart"
 
! Service
 
! Service
 
! Master
 
! Master
Line 17: Line 25:
 
|| PMTPA db*
 
|| PMTPA db*
 
|| PMTPA slave replication
 
|| PMTPA slave replication
−
|| KNAMS [[toolserver]] replication
+
|| EQIAD db slaves & snapshots / ESAMS [[toolserver]] replication
−
||  
+
|| http://noc.wikimedia.org/dbtree/
 +
|| 4 TB
 
|-
 
|-
 
|| External storage
 
|| External storage
 
|| PMTPA db*
 
|| PMTPA db*
 
|| PMTPA slave replication
 
|| PMTPA slave replication
−
|| KNAMS [[toolserver]] replication
+
|| EQIAD ES1004 replicates ms03; ES1001-3 is a copy of rc1 & cl1-cl21 /ESAMS [[toolserver]] replication
 
||  
 
||  
 +
|| 5 TB
 
|-
 
|-
 
|| Wiki data dumps
 
|| Wiki data dumps
−
|| PMTPA [[storage2]]
+
|| PMTPA [[dataset2]]
−
||  
+
|| Gluster storage (labstore1 under /publicdata-project directory) Last 5 good only.
−
|| [[Offsite_Backups#eBart|eBart]], [[Amazon Public Data Sets|AWS]]
+
|| EQIAD: [[dataset1001]] (via rsync). All public data: dumps.wikimedia.your.org. Last 5 good dumps: wikipedia.c3sl.ufpr.br (ftp/http/rsync).
−
|| Offsite syncs ad-hoc schedule - tfinc
+
|| See [[Dumps/Mirror status]] for more info.
−
|| 250GB
+
|| 6T (last 5 good)/ 29T (all)
 
|-
 
|-
 
|colspan=6|
 
|colspan=6|
Line 38: Line 48:
 
|-
 
|-
 
|| Uploads
 
|| Uploads
−
|| PMTPA [[ms1]]
+
|| PMTPA [[ms7]]
−
|| PMTPA [[ms5]]
+
|| PMTPA [[ms8]]
−
|| ESAMS [[ms6]] (partial)
+
|| EQIAD ms1002 / ESAMS [[ms6]] (partial)
−
|| In discussion on [[ms1]]
+
||  
−
|| 7T
+
|| 23T
 
|-
 
|-
 
|| Thumbs
 
|| Thumbs
−
|| PMTPA [[ms4]]
+
|| PMTPA [[ms5]]
−
||
+
 
||  
 
||  
 +
|| EQIAD ms1004
 
||
 
||
−
|| 2T
+
|| 12T
 
|-
 
|-
 
|colspan=6|
 
|colspan=6|
Line 57: Line 67:
 
|| MediaWiki config
 
|| MediaWiki config
 
|| PMTPA [[NFS]]
 
|| PMTPA [[NFS]]
 +
|| [[tridge]]
 +
|| Subversion
 +
||
 
||  
 
||  
−
|| OFFICE - brion
 
−
|| ad-hoc rsync
 
 
|-
 
|-
 
|| Apache config
 
|| Apache config
 
|| PMTPA [[NFS]]
 
|| PMTPA [[NFS]]
−
||  
+
|| [[tridge]]
 
|| Automatic
 
|| Automatic
 
|| rsync
 
|| rsync
 +
||
 
|-
 
|-
 
|| Subversion
 
|| Subversion
−
|| KNAMS [[mayflower]]
+
|| [[Formey]]
−
||  
+
|| [[tridge]]
 
||
 
||
 
||
 
||
 +
|| 3 GB
 +
|-
 +
|| Gerrit
 +
|| [[Manganese]]
 +
||
 +
|| [[Formey]]
 +
|| Databases on db9/10
 
|| 3 GB
 
|| 3 GB
 
|-
 
|-
 
|| [[Bugzilla]] data
 
|| [[Bugzilla]] data
 
|| PMTPA [[db9]]
 
|| PMTPA [[db9]]
−
||  
+
||[[db10]] replication
−
|| OFFICE - brion
+
||EQIAD [[db1008]]/[[db1025]] & snapshots
−
|| ad-hoc rsync
+
||
 
|| 1 GB
 
|| 1 GB
 
|-
 
|-
 
|| [[Bugzilla]] config
 
|| [[Bugzilla]] config
 
|| PMTPA [[db9]]
 
|| PMTPA [[db9]]
−
||  
+
||[[db10]] replication
−
|| OFFICE - brion
+
|| EQIAD [[db1008]]/[[db1025]] & snapshots
−
|| ad-hoc rsync
+
|| db10.pmtpa.w /a/sqldata  0 66532680 9020197  13.6  31:49 4724.1  31:49 4724.1
 
|| 1.5 GB
 
|| 1.5 GB
 
|-
 
|-
Line 98: Line 117:
 
|| linode
 
|| linode
 
||  
 
||  
−
|| OFFICE – tomasz; Amsterdam?
+
|| PMTPA [[tridge]]; ESAMS ?
−
|| ad-hoc rsync - is this automated?
+
||
 +
||
 
|-
 
|-
 
|colspan=6|
 
|colspan=6|
Line 107: Line 127:
 
|| [[OTRS]] data
 
|| [[OTRS]] data
 
|| PMTPA [[db9]]
 
|| PMTPA [[db9]]
−
||  
+
||[[db10]] replication
−
|class="missing"|  
+
||EQIAD [[db1008]]/[[db1025]] & snapshots
−
|| No Replication
+
||
 
|| 57 GB
 
|| 57 GB
 
|-
 
|-
Line 135: Line 155:
 
|| [[IMAP]] mail
 
|| [[IMAP]] mail
 
|| PMTPA [[sanger]]
 
|| PMTPA [[sanger]]
−
|| PMTPA [[mchenry]]
+
|| PMTPA [[mchenry]], Daily backups to [[tridge]]
 
|class="missing"|  
 
|class="missing"|  
 
|| see [[Mail#Backups]]
 
|| see [[Mail#Backups]]
Line 141: Line 161:
 
|-
 
|-
 
|| [[Mailing lists]]
 
|| [[Mailing lists]]
−
|| KNAMS [[lily]]
+
|| ESAMS [[lily]] - decommissioned
−
||  
+
|| EQIAD sodium
 
|| PMTPA [[mchenry]]
 
|| PMTPA [[mchenry]]
 
|| See [[Mailing lists#Backups]]
 
|| See [[Mailing lists#Backups]]
Line 149: Line 169:
 
|| [[DNS]]
 
|| [[DNS]]
 
|| PMTPA [[ns0]], [[ns1]]
 
|| PMTPA [[ns0]], [[ns1]]
−
||
+
|| checkin to sockpuppet SVN
 
|| ESAMS [[ns2]]
 
|| ESAMS [[ns2]]
 +
|| In SVN now
 
||  
 
||  
 
|-
 
|-
Line 158: Line 179:
 
|class="missing"|  
 
|class="missing"|  
 
|| ad-hoc downloads
 
|| ad-hoc downloads
 +
||
 
|-
 
|-
 
|colspan=6|
 
|colspan=6|
Line 165: Line 187:
 
|| Search databases
 
|| Search databases
 
|| PMTPA various
 
|| PMTPA various
 +
|| EQIAD is primary now
 +
|| SDTPA
 
||  
 
||  
−
|class="missing"|
 
−
|| regeneratable
 
 
|-
 
|-
 
|| HTTP logs
 
|| HTTP logs
 
|| PMTPA [[locke]]
 
|| PMTPA [[locke]]
−
||  
+
|| emery
−
|class="missing"|
+
 
|| /a/squid
 
|| /a/squid
 +
|| oxygen (eqiad)
 
|-
 
|-
 
|| MediaWiki logs
 
|| MediaWiki logs
 
|| PMTPA [[NFS]]
 
|| PMTPA [[NFS]]
−
||  
+
|| [[tridge]]
−
|class="missing"|  
+
|| [[fluorine]]
 
|| /home/wikipedia/logs
 
|| /home/wikipedia/logs
 +
||
 +
 
|-
 
|-
 
|colspan=6|
 
|colspan=6|
  
−
=== Misc data ===
+
=== Fundraising ===
 
|-
 
|-
 
|| Fundraising Front-End
 
|| Fundraising Front-End
−
|| PMTPA [[grosley]]
+
|| EQIAD [[aluminium]]
−
||
+
||  
−
|| Software now in SVN
+
|| software:svn<br>logs:[[storage3]]->[[tridge]]
−
|| [[tridge]] /data/civicrm-frontend/ Backs up once a week via a tar of the directory and rsync
+
|| puppet:files/misc/scripts/offhost_backups
−
|| 10 MB
+
|| 5GB
 
|-
 
|-
 
|| Fundraising Databases
 
|| Fundraising Databases
−
|| PMTPA [[db9]]
+
|| EQIAD [[db1008]]
−
||
+
|| [[db1025]] replication + snapshots
−
|class="missing"|
+
|| PMTPA [[storage3]] replication, snapshots, dumps<br>[[tridge]] daily rsync of dumps
−
|class="missing"|[[tridge]] /data/civicrm-data/ Backs up once a week with a database dump of the civicrm2 and drupal5 databases then rsync '''NOT UPDATED SINCE SEP 6 (srv10)'''
+
|| puppet:files/misc/scripts/dump_fundraisingdb
−
|| 160 MB
+
|| 1TB (15GB/day)
 +
|-
 +
|| Payment Processing
 +
|| software:svn<br>db+logs:PMTPA
 +
|| db:mysql replication<br>db+logs:encrypt+copy daily to [[tridge]]
 +
|| db+logs:replicate to [[silicon]]
 +
|| [https://office.wikimedia.org/wiki/Payments_cluster payments cluster]
 +
|| 100GB(?)
 +
|-
 +
|| Impression Logs
 +
|| PMTPA [[nas1]]
 +
|| 
 +
|| [[nas1001]]
 +
||
 +
|| 1.5TB year
 +
 
 +
|-
 +
|colspan=6|
 +
 
 +
=== Misc data ===
 
|-
 
|-
 
|| Office fileserver
 
|| Office fileserver
 
|| OFFICE [[fileserver]]
 
|| OFFICE [[fileserver]]
 +
|| weekly rsync to USB drive
 +
|class="missing"| rsync to [[tridge]] not running
 
||  
 
||  
−
|| tridge
 
−
|| not currently running
 
 
|| 2 GB(?)
 
|| 2 GB(?)
 
|-
 
|-
Line 210: Line 253:
 
||  
 
||  
 
|class="missing"|  
 
|class="missing"|  
−
|| currently no backups done
+
|| currently no backups done (can be enabled, if needed; MacOS only)
 
||
 
||
 
|-
 
|-
 
|| Office laptops
 
|| Office laptops
−
|| *
+
|| OFFICE *
 
||  
 
||  
 
|class="missing"|  
 
|class="missing"|  
−
|| Time Machine backups to usb drive on Imac (MacOS laptops)
+
|| Time Machine backups to usb drive on Imac (MacOS only)
 +
||
 
|-
 
|-
 
|| Server home directories
 
|| Server home directories
−
|| PMTPA [[db20]]
+
|| PMTPA nfs1 / nfs2
−
||
+
|| [[tridge]]
 
|class="missing"|  
 
|class="missing"|  
−
|| Weekly backups to Tridge
+
|| Daily backups to Tridge
 
|| 30 GB
 
|| 30 GB
 +
|-
 +
|| stat1 /a and /home
 +
|| -
 +
|| [[Amanda]] to [[tridge]]
 +
|class="missing"|
 +
|| [https://gerrit.wikimedia.org/r/#/c/9258/3 gerrit change]
 +
|| 400 GB
 
|}
 
|}
  
 
__NOTOC__
 
__NOTOC__
 
[[Category:Risk management]]
 
[[Category:Risk management]]
 +
[[Category:Dumps]]

Latest revision as of 21:43, 26 October 2012

This page is an inventory of our backups for both clusters data and office data.

Some definitions:

On-site
here means the backup is in the same physical location as the master, but on a different machine.
Off-site
means the backup is in a physically separate location. In most cases it's nice to have both!
Red cells
denote a lacking area which we want/need.

Contents


[edit] Service backup state

Service Master On-site Off-site Notes Approx. Space Needed

[edit] Wiki primary data

Wiki main databases PMTPA db* PMTPA slave replication EQIAD db slaves & snapshots / ESAMS toolserver replication http://noc.wikimedia.org/dbtree/ 4 TB
External storage PMTPA db* PMTPA slave replication EQIAD ES1004 replicates ms03; ES1001-3 is a copy of rc1 & cl1-cl21 /ESAMS toolserver replication 5 TB
Wiki data dumps PMTPA dataset2 Gluster storage (labstore1 under /publicdata-project directory) Last 5 good only. EQIAD: dataset1001 (via rsync). All public data: dumps.wikimedia.your.org. Last 5 good dumps: wikipedia.c3sl.ufpr.br (ftp/http/rsync). See Dumps/Mirror status for more info. 6T (last 5 good)/ 29T (all)

[edit] Images and media

Uploads PMTPA ms7 PMTPA ms8 EQIAD ms1002 / ESAMS ms6 (partial) 23T
Thumbs PMTPA ms5 EQIAD ms1004 12T

[edit] Software development and configuration

MediaWiki config PMTPA NFS tridge Subversion
Apache config PMTPA NFS tridge Automatic rsync
Subversion Formey tridge 3 GB
Gerrit Manganese Formey Databases on db9/10 3 GB
Bugzilla data PMTPA db9 db10 replication EQIAD db1008/db1025 & snapshots 1 GB
Bugzilla config PMTPA db9 db10 replication EQIAD db1008/db1025 & snapshots db10.pmtpa.w /a/sqldata 0 66532680 9020197 13.6 31:49 4724.1 31:49 4724.1 1.5 GB
Bugzilla frontend PMTPA isidore daily rsync to Tridge
Wikitech wiki linode PMTPA tridge; ESAMS ?

[edit] Communications

OTRS data PMTPA db9 db10 replication EQIAD db1008/db1025 & snapshots 57 GB
OTRS config PMTPA bart 19 GB
blogs frontend data PMTPA singer Daily backups to Tridge
bugzilla PMTPA isidore Daily backups to Tridge
IMAP mail PMTPA sanger PMTPA mchenry, Daily backups to tridge see Mail#Backups 32 GB
Mailing lists ESAMS lily - decommissioned EQIAD sodium PMTPA mchenry See Mailing lists#Backups 50 GB
DNS PMTPA ns0, ns1 checkin to sockpuppet SVN ESAMS ns2 In SVN now
Google docs Google :P ad-hoc downloads

[edit] Soft data

Search databases PMTPA various EQIAD is primary now SDTPA
HTTP logs PMTPA locke emery /a/squid oxygen (eqiad)
MediaWiki logs PMTPA NFS tridge fluorine /home/wikipedia/logs

[edit] Fundraising

Fundraising Front-End EQIAD aluminium software:svn
logs:storage3->tridge 		puppet:files/misc/scripts/offhost_backups 5GB
Fundraising Databases EQIAD db1008 db1025 replication + snapshots PMTPA storage3 replication, snapshots, dumps
tridge daily rsync of dumps 		puppet:files/misc/scripts/dump_fundraisingdb 1TB (15GB/day)
Payment Processing software:svn
db+logs:PMTPA 		db:mysql replication
db+logs:encrypt+copy daily to tridge 		db+logs:replicate to silicon payments cluster 100GB(?)
Impression Logs PMTPA nas1 nas1001 1.5TB year

[edit] Misc data

Office fileserver OFFICE fileserver weekly rsync to USB drive rsync to tridge not running 2 GB(?)
Office workstations OFFICE * currently no backups done (can be enabled, if needed; MacOS only)
Office laptops OFFICE * Time Machine backups to usb drive on Imac (MacOS only)
Server home directories PMTPA nfs1 / nfs2 tridge Daily backups to Tridge 30 GB
stat1 /a and /home - Amanda to tridge gerrit change 400 GB
Retrieved from "http://wikitech-old.wikimedia.org/index.php?title=Backup_procedures&oldid=52868"
Personal tools
Namespaces

Variants
Actions
Navigation
Ops documentation
Wiki
Toolbox