Backup procedures

From Wikitech
(Difference between revisions)
Jump to: navigation, search
(→‎Communications)
(→‎Software development and configuration)
 
(135 intermediate revisions by 19 users not shown)
Line 1: Line 1:
 +
This page is an inventory of our backups for both clusters data and office data.
 +
 +
Some definitions:
 +
 +
; On-site: here means the backup is in the same physical location as the master, but on a different machine.
 +
; Off-site: means the backup is in a physically separate location. In most cases it's nice to have both!
 +
; Red cells: denote a lacking area which we want/need.
 +
 +
__TOC__
 +
 
== Service backup state ==
 
== Service backup state ==
  
−
{| border="1" class="backup-chart"
+
{| border="1" class="wikitable backup-chart"
 
! Service
 
! Service
 
! Master
 
! Master
Line 7: Line 17:
 
! Off-site
 
! Off-site
 
! Notes
 
! Notes
 +
! Approx. Space Needed
 
|-
 
|-
−
|colspan=5|
+
|colspan=6|
 
=== Wiki primary data ===
 
=== Wiki primary data ===
 
|-
 
|-
Line 14: Line 25:
 
|| PMTPA db*
 
|| PMTPA db*
 
|| PMTPA slave replication
 
|| PMTPA slave replication
−
|| KNAMS [[toolserver]] replication
+
|| EQIAD db slaves & snapshots / ESAMS [[toolserver]] replication
−
||  
+
|| http://noc.wikimedia.org/dbtree/
 +
|| 4 TB
 
|-
 
|-
 
|| External storage
 
|| External storage
 
|| PMTPA db*
 
|| PMTPA db*
 
|| PMTPA slave replication
 
|| PMTPA slave replication
−
|| KNAMS [[toolserver]] replication
+
|| EQIAD ES1004 replicates ms03; ES1001-3 is a copy of rc1 & cl1-cl21 /ESAMS [[toolserver]] replication
 
||  
 
||  
 +
|| 5 TB
 
|-
 
|-
 
|| Wiki data dumps
 
|| Wiki data dumps
−
|| PMTPA [[benet]]
+
|| PMTPA [[dataset2]]
−
||  
+
|| Gluster storage (labstore1 under /publicdata-project directory) Last 5 good only.
−
|class="missing"|  
+
|| EQIAD: [[dataset1001]] (via rsync). All public data: dumps.wikimedia.your.org. Last 5 good dumps: wikipedia.c3sl.ufpr.br (ftp/http/rsync).
−
|| some data files on storage2
+
|| See [[Dumps/Mirror status]] for more info.
 +
|| 6T (last 5 good)/ 29T (all)
 
|-
 
|-
−
|colspan=5|
+
|colspan=6|
  
 
=== Images and media ===
 
=== Images and media ===
 
|-
 
|-
 
|| Uploads
 
|| Uploads
−
|| PMTPA [[amane]]
+
|| PMTPA [[ms7]]
−
|| PMTPA [[storage2]] – rsync
+
|| PMTPA [[ms8]]
−
|class="missing"|  
+
|| EQIAD ms1002 / ESAMS [[ms6]] (partial)
−
|| rsync to storage2 needs automation; planning rsync to KNAMS as well
+
||  
 +
|| 23T
 
|-
 
|-
−
|| Upload dumps
+
|| Thumbs
−
|| n/a
+
|| PMTPA [[ms5]]
−
||
+
−
|class="missing"|
+
 
||  
 
||  
 +
|| EQIAD ms1004
 +
||
 +
|| 12T
 
|-
 
|-
−
|colspan=5|
+
|colspan=6|
 +
 
 
=== Software development and configuration ===
 
=== Software development and configuration ===
 
|-
 
|-
 
|| MediaWiki config
 
|| MediaWiki config
−
|| PMTPA [[suda]]
+
|| PMTPA [[NFS]]
−
||  
+
|| [[tridge]]
−
|class="missing"|  
+
|| Subversion
 +
||
 
||  
 
||  
 
|-
 
|-
 
|| Apache config
 
|| Apache config
−
|| PMTPA [[suda]]
+
|| PMTPA [[NFS]]
−
||  
+
|| [[tridge]]
−
|class="missing"|  
+
|| Automatic
 +
|| rsync
 
||  
 
||  
 
|-
 
|-
 
|| Subversion
 
|| Subversion
−
|| KNAMS [[mayflower]]
+
|| [[Formey]]
−
||  
+
|| [[tridge]]
−
|| OFFICE – brion-imac
+
||
−
|| ad-hoc rsync
+
||
 +
|| 3 GB
 
|-
 
|-
−
|| Bugzilla data
+
|| Gerrit
−
|| PMTPA [[srv7]]
+
|| [[Manganese]]
 
||  
 
||  
−
|class="missing"|  
+
|| [[Formey]]
−
|| srv8 slave replication is BROKEN
+
|| Databases on db9/10
 +
|| 3 GB
 
|-
 
|-
−
|| Bugzilla config
+
|| [[Bugzilla]] data
 +
|| PMTPA [[db9]]
 +
||[[db10]] replication
 +
||EQIAD [[db1008]]/[[db1025]] & snapshots
 +
||
 +
|| 1 GB
 +
|-
 +
|| [[Bugzilla]] config
 +
|| PMTPA [[db9]]
 +
||[[db10]] replication
 +
|| EQIAD [[db1008]]/[[db1025]] & snapshots
 +
|| db10.pmtpa.w /a/sqldata  0 66532680 9020197  13.6  31:49 4724.1  31:49 4724.1
 +
|| 1.5 GB
 +
|-
 +
|| [[Bugzilla]] frontend
 
|| PMTPA [[isidore]]
 
|| PMTPA [[isidore]]
 +
|| daily rsync to [[Tridge]]
 +
||
 
||  
 
||  
−
|class="missing"|
 
 
||  
 
||  
 
|-
 
|-
 
|| Wikitech wiki
 
|| Wikitech wiki
−
|| leuksman.com
+
|| linode
 +
||
 +
|| PMTPA [[tridge]]; ESAMS ?
 +
||
 
||  
 
||  
−
|| OFFICE – brion-imac
 
−
|| ad-hoc rsync
 
 
|-
 
|-
−
|colspan=5|
+
|colspan=6|
  
 
=== Communications ===
 
=== Communications ===
 
|-
 
|-
 
|| [[OTRS]] data
 
|| [[OTRS]] data
−
|| PMTPA [[srv7]]
+
|| PMTPA [[db9]]
−
||  
+
||[[db10]] replication
−
|class="missing"|  
+
||EQIAD [[db1008]]/[[db1025]] & snapshots
−
|| srv8 slave replication is BROKEN
+
||
 +
|| 57 GB
 
|-
 
|-
 
|| [[OTRS]] config
 
|| [[OTRS]] config
Line 98: Line 136:
 
||  
 
||  
 
|class="missing"|  
 
|class="missing"|  
 +
||
 +
|| 19 GB
 +
|-
 +
|| blogs frontend data
 +
|| PMTPA [[singer]]
 +
|| Daily backups to [[Tridge]]
 +
|class="missing"|
 +
||
 +
||
 +
|-
 +
|| bugzilla
 +
|| PMTPA [[isidore]]
 +
|| Daily backups to [[Tridge]]
 +
|class="missing"|
 +
||
 
||  
 
||  
 
|-
 
|-
 
|| [[IMAP]] mail
 
|| [[IMAP]] mail
 
|| PMTPA [[sanger]]
 
|| PMTPA [[sanger]]
−
|| PMTPA [[mchenry]]
+
|| PMTPA [[mchenry]], Daily backups to [[tridge]]
 
|class="missing"|  
 
|class="missing"|  
 
|| see [[Mail#Backups]]
 
|| see [[Mail#Backups]]
 +
|| 32 GB
 
|-
 
|-
 
|| [[Mailing lists]]
 
|| [[Mailing lists]]
−
|| KNAMS [[lily]]
+
|| ESAMS [[lily]] - decommissioned
−
||  
+
|| EQIAD sodium
 
|| PMTPA [[mchenry]]
 
|| PMTPA [[mchenry]]
 
|| See [[Mailing lists#Backups]]
 
|| See [[Mailing lists#Backups]]
 +
|| 50 GB
 
|-
 
|-
 
|| [[DNS]]
 
|| [[DNS]]
−
|| PMTPA [[ns0]]
+
|| PMTPA [[ns0]], [[ns1]]
−
||
+
|| checkin to sockpuppet SVN
−
|| KNAMS [[n1]], [[ns2]]
+
|| ESAMS [[ns2]]
 +
|| In SVN now
 
||  
 
||  
−
|-
 
−
|| [[CRM]]
 
−
|| PMTPA [[friedrich]]
 
−
||
 
−
|| OFFICE brion-imac
 
−
|| ad-hoc syncs
 
 
|-
 
|-
 
|| [[Google docs]]
 
|| [[Google docs]]
Line 129: Line 179:
 
|class="missing"|  
 
|class="missing"|  
 
|| ad-hoc downloads
 
|| ad-hoc downloads
 +
||
 
|-
 
|-
−
|colspan=5|
+
|colspan=6|
  
 
=== Soft data ===
 
=== Soft data ===
Line 136: Line 187:
 
|| Search databases
 
|| Search databases
 
|| PMTPA various
 
|| PMTPA various
 +
|| EQIAD is primary now
 +
|| SDTPA
 
||  
 
||  
−
|class="missing"|
 
−
|| regeneratable
 
−
|-
 
−
|| Prefix search
 
−
|| PMTPA [[yongle]]
 
−
||
 
−
|class="missing"|
 
−
|| regeneratable
 
 
|-
 
|-
 
|| HTTP logs
 
|| HTTP logs
−
|| YASEO [[henbane]]
+
|| PMTPA [[locke]]
−
||  
+
|| emery
−
|class="missing"|
+
|| /a/squid
−
|| /a/logs
+
|| oxygen (eqiad)
 
|-
 
|-
 
|| MediaWiki logs
 
|| MediaWiki logs
−
|| PMTPA [[suda]]
+
|| PMTPA [[NFS]]
−
||  
+
|| [[tridge]]
−
|class="missing"|  
+
|| [[fluorine]]
 
|| /home/wikipedia/logs
 
|| /home/wikipedia/logs
 +
||
 +
 
|-
 
|-
−
|colspan=5|
+
|colspan=6|
 +
 
 +
=== Fundraising ===
 +
|-
 +
|| Fundraising Front-End
 +
|| EQIAD [[aluminium]]
 +
||
 +
|| software:svn<br>logs:[[storage3]]->[[tridge]]
 +
|| puppet:files/misc/scripts/offhost_backups
 +
|| 5GB
 +
|-
 +
|| Fundraising Databases
 +
|| EQIAD [[db1008]]
 +
|| [[db1025]] replication + snapshots
 +
|| PMTPA [[storage3]] replication, snapshots, dumps<br>[[tridge]] daily rsync of dumps
 +
|| puppet:files/misc/scripts/dump_fundraisingdb
 +
|| 1TB (15GB/day)
 +
|-
 +
|| Payment Processing
 +
|| software:svn<br>db+logs:PMTPA
 +
|| db:mysql replication<br>db+logs:encrypt+copy daily to [[tridge]]
 +
|| db+logs:replicate to [[silicon]]
 +
|| [https://office.wikimedia.org/wiki/Payments_cluster payments cluster]
 +
|| 100GB(?)
 +
|-
 +
|| Impression Logs
 +
|| PMTPA [[nas1]]
 +
|| 
 +
|| [[nas1001]]
 +
||
 +
|| 1.5TB year
 +
 
 +
|-
 +
|colspan=6|
 +
 
 
=== Misc data ===
 
=== Misc data ===
 
|-
 
|-
 
|| Office fileserver
 
|| Office fileserver
 
|| OFFICE [[fileserver]]
 
|| OFFICE [[fileserver]]
 +
|| weekly rsync to USB drive
 +
|class="missing"| rsync to [[tridge]] not running
 
||  
 
||  
−
|class="missing"|  
+
|| 2 GB(?)
−
|| meant to be a synced copy in PMTPA and maybe elsewhere?
+
 
|-
 
|-
 
|| Office workstations
 
|| Office workstations
Line 171: Line 253:
 
||  
 
||  
 
|class="missing"|  
 
|class="missing"|  
−
|| currently no backups done
+
|| currently no backups done (can be enabled, if needed; MacOS only)
 +
||
 
|-
 
|-
 
|| Office laptops
 
|| Office laptops
−
|| *
+
|| OFFICE *
 
||  
 
||  
 
|class="missing"|  
 
|class="missing"|  
−
|| currently no centralized backups done
+
|| Time Machine backups to usb drive on Imac (MacOS only)
 +
||
 
|-
 
|-
 
|| Server home directories
 
|| Server home directories
−
|| PMTPA [[suda]]
+
|| PMTPA nfs1 / nfs2
−
||  
+
|| [[tridge]]
 +
|class="missing"|  
 +
|| Daily backups to Tridge
 +
|| 30 GB
 +
|-
 +
|| stat1 /a and /home
 +
|| -
 +
|| [[Amanda]] to [[tridge]]
 
|class="missing"|  
 
|class="missing"|  
−
|| Some ad-hoc duplication somewhere... some personal copies
+
|| [https://gerrit.wikimedia.org/r/#/c/9258/3 gerrit change]
 +
|| 400 GB
 
|}
 
|}
  
 
__NOTOC__
 
__NOTOC__
 +
[[Category:Risk management]]
 +
[[Category:Dumps]]

Latest revision as of 21:43, 26 October 2012

This page is an inventory of our backups for both clusters data and office data.

Some definitions:

On-site
here means the backup is in the same physical location as the master, but on a different machine.
Off-site
means the backup is in a physically separate location. In most cases it's nice to have both!
Red cells
denote a lacking area which we want/need.

Contents


[edit] Service backup state

Service Master On-site Off-site Notes Approx. Space Needed

[edit] Wiki primary data

Wiki main databases PMTPA db* PMTPA slave replication EQIAD db slaves & snapshots / ESAMS toolserver replication http://noc.wikimedia.org/dbtree/ 4 TB
External storage PMTPA db* PMTPA slave replication EQIAD ES1004 replicates ms03; ES1001-3 is a copy of rc1 & cl1-cl21 /ESAMS toolserver replication 5 TB
Wiki data dumps PMTPA dataset2 Gluster storage (labstore1 under /publicdata-project directory) Last 5 good only. EQIAD: dataset1001 (via rsync). All public data: dumps.wikimedia.your.org. Last 5 good dumps: wikipedia.c3sl.ufpr.br (ftp/http/rsync). See Dumps/Mirror status for more info. 6T (last 5 good)/ 29T (all)

[edit] Images and media

Uploads PMTPA ms7 PMTPA ms8 EQIAD ms1002 / ESAMS ms6 (partial) 23T
Thumbs PMTPA ms5 EQIAD ms1004 12T

[edit] Software development and configuration

MediaWiki config PMTPA NFS tridge Subversion
Apache config PMTPA NFS tridge Automatic rsync
Subversion Formey tridge 3 GB
Gerrit Manganese Formey Databases on db9/10 3 GB
Bugzilla data PMTPA db9 db10 replication EQIAD db1008/db1025 & snapshots 1 GB
Bugzilla config PMTPA db9 db10 replication EQIAD db1008/db1025 & snapshots db10.pmtpa.w /a/sqldata 0 66532680 9020197 13.6 31:49 4724.1 31:49 4724.1 1.5 GB
Bugzilla frontend PMTPA isidore daily rsync to Tridge
Wikitech wiki linode PMTPA tridge; ESAMS ?

[edit] Communications

OTRS data PMTPA db9 db10 replication EQIAD db1008/db1025 & snapshots 57 GB
OTRS config PMTPA bart 19 GB
blogs frontend data PMTPA singer Daily backups to Tridge
bugzilla PMTPA isidore Daily backups to Tridge
IMAP mail PMTPA sanger PMTPA mchenry, Daily backups to tridge see Mail#Backups 32 GB
Mailing lists ESAMS lily - decommissioned EQIAD sodium PMTPA mchenry See Mailing lists#Backups 50 GB
DNS PMTPA ns0, ns1 checkin to sockpuppet SVN ESAMS ns2 In SVN now
Google docs Google :P ad-hoc downloads

[edit] Soft data

Search databases PMTPA various EQIAD is primary now SDTPA
HTTP logs PMTPA locke emery /a/squid oxygen (eqiad)
MediaWiki logs PMTPA NFS tridge fluorine /home/wikipedia/logs

[edit] Fundraising

Fundraising Front-End EQIAD aluminium software:svn
logs:storage3->tridge 		puppet:files/misc/scripts/offhost_backups 5GB
Fundraising Databases EQIAD db1008 db1025 replication + snapshots PMTPA storage3 replication, snapshots, dumps
tridge daily rsync of dumps 		puppet:files/misc/scripts/dump_fundraisingdb 1TB (15GB/day)
Payment Processing software:svn
db+logs:PMTPA 		db:mysql replication
db+logs:encrypt+copy daily to tridge 		db+logs:replicate to silicon payments cluster 100GB(?)
Impression Logs PMTPA nas1 nas1001 1.5TB year

[edit] Misc data

Office fileserver OFFICE fileserver weekly rsync to USB drive rsync to tridge not running 2 GB(?)
Office workstations OFFICE * currently no backups done (can be enabled, if needed; MacOS only)
Office laptops OFFICE * Time Machine backups to usb drive on Imac (MacOS only)
Server home directories PMTPA nfs1 / nfs2 tridge Daily backups to Tridge 30 GB
stat1 /a and /home - Amanda to tridge gerrit change 400 GB
Retrieved from "http://wikitech-old.wikimedia.org/index.php?title=Backup_procedures&oldid=52868"
Personal tools
Namespaces

Variants
Actions
Navigation
Ops documentation
Wiki
Toolbox