Db20 to NFS1-2 migration plan

From Wikitech
(Difference between revisions)
Jump to: navigation, search
(→‎New drbd.conf: new IPs)
(recat)
 
(4 intermediate revisions by one user not shown)
Line 13: Line 13:
  
 
=== Preparation ===
 
=== Preparation ===
−
* Prepare nfs2 for drbd
+
* Check DRAC access
 +
* Prepare nfs1 for drbd
 
* Add DNS entries for new internal IPs
 
* Add DNS entries for new internal IPs
 
* Set up cross link networking between nfs1/nfs2
 
* Set up cross link networking between nfs1/nfs2
Line 19: Line 20:
 
** replication rate 110M
 
** replication rate 110M
 
* Write final configuration files for nfs1/nfs2
 
* Write final configuration files for nfs1/nfs2
−
* Install /etc/exports on nfs1/nfs2
+
* Install <tt>/etc/exports</tt> on nfs1/nfs2
 
* Install NFS kernel server on nfs1/nfs2
 
* Install NFS kernel server on nfs1/nfs2
  
Line 49: Line 50:
 
disk /dev/mapper/mirror-nfshome;
 
disk /dev/mapper/mirror-nfshome;
 
address 10.3.0.1:7789;
 
address 10.3.0.1:7789;
−
meta-disk-internal;
+
meta-disk internal;
 
}
 
}
  
Line 64: Line 65:
  
 
'''On nfs1''':
 
'''On nfs1''':
−
* Install new /etc/drbd.conf
+
* Install new <tt>/etc/drbd.conf</tt>
 
* Create device meta-data:
 
* Create device meta-data:
 
  drbdadm create-md nfshome
 
  drbdadm create-md nfshome
Line 73: Line 74:
 
'''On db20''':
 
'''On db20''':
 
* Stop NFS kernel server
 
* Stop NFS kernel server
−
* Remove NFS service IP 10.0.5.8:
+
* Remove NFS service IP <tt>10.0.5.8</tt>:
 
  ip addr del 10.0.5.8/32 dev eth0
 
  ip addr del 10.0.5.8/32 dev eth0
−
* Stop userland processes using /home (fuser -m)
+
* Stop userland processes using <tt>/home</tt> (<tt>fuser -m</tt>)
−
* Unmount /home
+
* Unmount <tt>/home</tt>
−
* Comment /home mount in /etc/fstab
+
* Comment <tt>/home</tt> mount in <tt>/etc/fstab</tt>
−
* Check consistency of nfshome replication (cat /proc/drbdadm)
+
* Check consistency of nfshome replication (<tt>cat /proc/drbdadm</tt>)
 
* Disable resource:
 
* Disable resource:
 
  drbdadm down nfshome
 
  drbdadm down nfshome
Line 87: Line 88:
  
 
'''On nfs2''':
 
'''On nfs2''':
−
* Check consistency of nfshome replication (cat /proc/drbdadm)
+
* Check consistency of nfshome replication (<tt>cat /proc/drbdadm</tt>)
 
* Disable resource:
 
* Disable resource:
 
  drbdadm down nfshome
 
  drbdadm down nfshome
 
* Save the meta-data in a text file:
 
* Save the meta-data in a text file:
 
  drbdadm dump-md nfshome > /root/db20-nfs2-drbd.metadata
 
  drbdadm dump-md nfshome > /root/db20-nfs2-drbd.metadata
−
* Install new /etc/drbd.conf
+
* Install new <tt>/etc/drbd.conf</tt>
 
* Create device meta-data:
 
* Create device meta-data:
 
  drbdadm create-md nfshome
 
  drbdadm create-md nfshome
Line 108: Line 109:
 
  drbdadm -- --overwrite-data-of-peer primary nfshome
 
  drbdadm -- --overwrite-data-of-peer primary nfshome
 
* Check replication status (cat /proc/drbd)
 
* Check replication status (cat /proc/drbd)
−
* Add /etc/fstab entry:
+
* Add <tt>/etc/fstab</tt> entry:
 
  /dev/drbd1 /home          jfs    defaults        0      2
 
  /dev/drbd1 /home          jfs    defaults        0      2
−
* Mount /home:
+
* Mount <tt>/home</tt>:
 
  mount /home
 
  mount /home
−
* Migrate NFS service IP 10.0.5.8 to nfs2:
+
* Migrate NFS service IP <tt>10.0.5.8</tt> to nfs2:
 
  ip addr add 10.0.5.8/32 dev eth0
 
  ip addr add 10.0.5.8/32 dev eth0
−
* Configure service IP on nfs2 on bootup. Add entry to /etc/networking/interfaces eth0:
+
* Configure service IP on nfs2 on bootup. Add entry to <tt>/etc/networking/interfaces</tt> eth0:
 
  up ip addr add 10.0.5.8/32 dev $IFACE
 
  up ip addr add 10.0.5.8/32 dev $IFACE
 
* Start NFS kernel server
 
* Start NFS kernel server
Line 122: Line 123:
 
* Write documentation on wikitech about the DRBD setup
 
* Write documentation on wikitech about the DRBD setup
  
−
[[Category:Migration plans]] [[Category:NFS]] [[Category:DRBD]]
+
=== Rollback ===
 +
If anything goes wrong, the filesystem on <tt>/dev/drbd1</tt>, or alternatively <tt>/dev/sda6</tt> should be in tact and can be mounted. NFS service can then be restored quickly.
 +
 
 +
[[Category:Migration plans]]

Latest revision as of 00:08, 21 February 2012

NFS home DRBD migration plan.

Contents

[edit] Hosts

  • db20 - old server
  • nfs1 - fresh host
  • nfs2 - host replicating from db20

[edit] Differences

  • Move replication from db20->nfs2 to nfs2->nfs1
  • Use protocol C, not A

[edit] Plan

[edit] Preparation

  • Check DRAC access
  • Prepare nfs1 for drbd
  • Add DNS entries for new internal IPs
  • Set up cross link networking between nfs1/nfs2
    • Protocol C, not A
    • replication rate 110M
  • Write final configuration files for nfs1/nfs2
  • Install /etc/exports on nfs1/nfs2
  • Install NFS kernel server on nfs1/nfs2

[edit] IP plan 

; Cross link between nfs1/nfs2: 10.3.0.0/30

1       1H      IN PTR  nfs1-xlink0.pmtpa.wmnet.
2       1H      IN PTR  nfs2-xlink0.pmtpa.wmnet.

nfs1-xlink0     1H      IN A            10.3.0.1
nfs2-xlink0     1H      IN A            10.3.0.2

[edit] New drbd.conf 

# drbd.conf for replication between nfs1 and nfs2
# Written on 2009/11/23 by Mark Bergsma <mark@wikimedia.org>

global {
	usage-count no;
}

common {
	protocol C;
}

resource nfshome {
	on nfs1 {
		device /dev/drbd1;
		disk /dev/mapper/mirror-nfshome;
		address 10.3.0.1:7789;
		meta-disk internal;
	}

	on nfs2 {
		device /dev/drbd1;
		disk /dev/mapper/mirror-nfshome;
		address 10.3.0.2:7789;
		meta-disk internal;
	}
}

[edit] Migration

On nfs1:

  • Install new /etc/drbd.conf
  • Create device meta-data: 
drbdadm create-md nfshome
  • Attach and setup new resource, but do not connect yet: 
drbdadm attach nfshome
drbdadm syncer nfshome

On db20:

  • Stop NFS kernel server
  • Remove NFS service IP 10.0.5.8: 
ip addr del 10.0.5.8/32 dev eth0
  • Stop userland processes using /home (fuser -m)
  • Unmount /home
  • Comment /home mount in /etc/fstab
  • Check consistency of nfshome replication (cat /proc/drbdadm)
  • Disable resource: 
drbdadm down nfshome
  • Save the meta-data in a text file: 
drbdadm dump-md nfshome > /root/db20-nfs2-drbd.metadata

nfs2 should now have an exact block level replica of the /home filesystem

On nfs2:

  • Check consistency of nfshome replication (cat /proc/drbdadm)
  • Disable resource: 
drbdadm down nfshome
  • Save the meta-data in a text file: 
drbdadm dump-md nfshome > /root/db20-nfs2-drbd.metadata
  • Install new /etc/drbd.conf
  • Create device meta-data: 
drbdadm create-md nfshome
  • Enable the new resource: 
drbdadm attach nfshome
drbdadm syncer nfshome
drbdadm connect nfshome

On nfs1:

  • Connect the new resource: 
drbdadm connect nfshome

On nfs2:

  • Start initial device synchronisation (ON NFS2 ONLY!): 
drbdadm -- --overwrite-data-of-peer primary nfshome
  • Check replication status (cat /proc/drbd)
  • Add /etc/fstab entry: 
/dev/drbd1	/home           jfs     defaults        0       2
  • Mount /home: 
mount /home
  • Migrate NFS service IP 10.0.5.8 to nfs2: 
ip addr add 10.0.5.8/32 dev eth0
  • Configure service IP on nfs2 on bootup. Add entry to /etc/networking/interfaces eth0: 
up ip addr add 10.0.5.8/32 dev $IFACE
  • Start NFS kernel server

[edit] Wrap up

  • Migrate udp2log, /usr/local/bin/demux.py from db20 to nfs1 and nfs2
  • Write documentation on wikitech about the DRBD setup

[edit] Rollback

If anything goes wrong, the filesystem on /dev/drbd1, or alternatively /dev/sda6 should be in tact and can be mounted. NFS service can then be restored quickly.

Retrieved from "http://wikitech-old.wikimedia.org/index.php?title=Db20_to_NFS1-2_migration_plan&oldid=43590"
Personal tools
Namespaces

Variants
Actions
Navigation
Ops documentation
Wiki
Toolbox