Db20 to NFS1-2 migration plan

Revision as of 17:21, 23 November 2009

NFS home DRBD migration plan.

Hosts

• db20 - old server
• nfs1 - fresh host
• nfs2 - host replicating from db20

Differences

• Move replication from db20->nfs2 to nfs2->nfs1
• Use protocol C, not A

Plan

Preparation

• Prepare nfs2 for drbd
• Add DNS entries for new internal IPs
• Set up cross link networking between nfs1/nfs2
  • Protocol C, not A
  • replication rate 110M
• Write final configuration files for nfs1/nfs2
• Install /etc/exports on nfs1/nfs2
• Install NFS kernel server on nfs1/nfs2

IP plan

; Cross link between nfs1/nfs2: 10.3.0.0/30

1       1H      IN PTR  nfs1-xlink0.pmtpa.wmnet.
2       1H      IN PTR  nfs2-xlink0.pmtpa.wmnet.

nfs1-xlink0     1H      IN A            10.3.0.1
nfs2-xlink0     1H      IN A            10.3.0.2

New drbd.conf

# drbd.conf for replication between nfs1 and nfs2
# Written on 2009/11/23 by Mark Bergsma <mark@wikimedia.org>

global {
	usage-count no;
}

common {
	protocol C;
}

resource nfshome {
	on nfs1 {
		device /dev/drbd1;
		disk /dev/mapper/mirror-nfshome;
		address 10.3.0.1:7789;
		meta-disk-internal;
	}

	on nfs2 {
		device /dev/drbd1;
		disk /dev/mapper/mirror-nfshome;
		address 10.3.0.2:7789;
		meta-disk internal;
	}
}

Migration

On nfs1:

• Install new /etc/drbd.conf
• Create device meta-data:

drbdadm create-md nfshome

• Attach and setup new resource, but do not connect yet:

drbdadm attach nfshome
drbdadm syncer nfshome

On db20:

• Stop NFS kernel server
• Remove NFS service IP 10.0.5.8:

ip addr del 10.0.5.8/32 dev eth0

• Stop userland processes using /home (fuser -m)
• Unmount /home
• Comment /home mount in /etc/fstab
• Check consistency of nfshome replication (cat /proc/drbdadm)
• Disable resource:

drbdadm down nfshome

• Save the meta-data in a text file:

drbdadm dump-md nfshome > /root/db20-nfs2-drbd.metadata

nfs2 should now have an exact block level replica of the /home filesystem

On nfs2:

• Check consistency of nfshome replication (cat /proc/drbdadm)
• Disable resource:

drbdadm down nfshome

• Save the meta-data in a text file:

drbdadm dump-md nfshome > /root/db20-nfs2-drbd.metadata

• Install new /etc/drbd.conf
• Create device meta-data:

drbdadm create-md nfshome

• Enable the new resource:

drbdadm attach nfshome
drbdadm syncer nfshome
drbdadm connect nfshome

On nfs1:

• Connect the new resource:

drbdadm connect nfshome

On nfs2:

• Start initial device synchronisation (ON NFS2 ONLY!):

drbdadm -- --overwrite-data-of-peer primary nfshome

• Check replication status (cat /proc/drbd)
• Add /etc/fstab entry:

/dev/drbd1	/home           jfs     defaults        0       2

• Mount /home:

mount /home

• Migrate NFS service IP 10.0.5.8 to nfs2:

ip addr add 10.0.5.8/32 dev eth0

• Configure service IP on nfs2 on bootup. Add entry to /etc/networking/interfaces eth0:

up ip addr add 10.0.5.8/32 dev $IFACE

• Start NFS kernel server

Wrap up

• Migrate udp2log, /usr/local/bin/demux.py from db20 to nfs1 and nfs2
• Write documentation on wikitech about the DRBD setup

Rollback

If anything goes wrong, the filesystem on /dev/drbd1, or alternatively /dev/sda6 should be in tact and can be mounted. NFS service can then be restored quickly.