Swift/Deploy Plan - R720xds in tampa

Revision as of 18:22, 7 February 2013

Draft, edit at will

Current status

• Currently replaced: ms-be6,7,8,10
• Host which has fallen over a few times recently: ms-be3
• Hosts with disk issues (so they should be replaced next): ms-be5,11,12
• Other hosts with ssds: ms-be9
• The rest: ms-be1,2,4

Proposed schedule

Bah, the last proposed schedule died a horrible death; it was taking 5 days for one complete run of the object replicators. After some back of the napkin calculations, here's a new schedule (old one is hidden in html comments). I'll be monitoring to see if the object replication runs actually finish in two days (nope. 3.5 days) when we move this much data around; if they do, we'll try the below:

• Mon Dec 10 -- done

take out the two bad rings (remove, not weight 0):

d321 ms-be10 (10.0.6.209) sdc,

d294 ms-be7 (10.0.6.206) sda (which was replaced but we can't get linux to see it so whatever)

add weight to ms-be8 (10.0.6.207) to 100

add weight to ms-be10 (10.0.6.209) to 100

• Fri Dec 14 -- done

remove weight from ms-be5 (10.0.6.204) to 66

add weight to ms-be7 (10.0.6.206) to 100

• Fri Dec 21 -- done

remove weight from ms-be5 to 33

put in new ms-be3 to 33

• Mon Dec 31 -- done

remove weight from ms-be5 to 0

add weight to ms-be3 to 66

• Tue Jan 8 - done

power off ms-be5

remove weight from ms-be11 to 66

put in new ms-be5 at 33

• Mon Jan 13 - done

remove weight from ms-be11 to 33

add weight to ms-be3 to 100

• Mon Jan 21 - done

remove weight from ms-be11 to 0

add weight to ms-be5 at 66

• Mon Jan 28 - done

remove weight from ms-be12 to 66

power off ms-be11

add weight to ms-be5 at 100

• Thur Feb 7 - done

remove weight from ms-be12 to 33

put in new ms-be11 at 33

• Mon Jan 7 - from here on dates need to be recalculated

remove weight from ms-be12 to 0

add weight to ms-be11 at 66

• Wed Jan 9

remove weight from me-be9 to 66

power off ms-be12

add weight to ms-be11 at 100

• Fri Jan 11

remove weight from ms-be9 to 33

put in new ms-be12 to 33

• Mon Jan 7 -- dates from here on need t be recalculated

remove weight from ms-be9 to 0

add weight to ms-be12 to 66

• Wed Jan 9

remove weight from ms-be2 to 66

power off ms-be9

add weight to ms-be12 to 100

Note: ms-be9,1,2,4 go to ceph testing

• Fri Jan 11

remove weight from ms-be2 to 33

• Mon Jan 14

remove weight from ms-be2 to 0

• Wed Jan 16

remove weight from ms-be1 to 66

power off ms-be2

• Fri Jan 18

remove weight from ms-be1 to 33

• Mon Jan 21

remove weight from ms-be1 to 0

• Wed Jan 23

remove weight from ms-be4 to 66

power off ms-be1

• Fri Jan 25

remove weight from ms-be4 to 33

• Mon Jan 28

remove weight from ms-be4 to 0

• Web Jan 31

power off ms-be4

This schedule will be adjusted (shorter or longer) as we see how the cluster behaves.

Note: all hosts get ssds

Logistics

• What is our deadline for returning the C2100s?
• Do we have enough ssds to put in all remaining 720xds? If not let's get them (and what's the turnaround time for those?) YES we do.