Swift/Deploy Plan - R720xds in tampa

Draft, edit at will

Current status

• Currently replaced: ms-be6,7,8,10
• Host which has fallen over a few times recently: ms-be3
• Hosts with disk issues (so they should be replaced next): ms-be5,11,12
• Other hosts with ssds: ms-be9
• The rest: ms-be1,2,4

Proposed schedule

Bah, the last proposed schedule died a horrible death; it was taking 5 days for one complete run of the object replicators. After some back of the napkin calculations, here's a new schedule (old one is hidden in html comments). I'll be monitoring to see if the object replication runs actually finish in two days (nope. 3.5 days) when we move this much data around; if they do, we'll try the below:

• Mon Dec 10 -- done

take out the two bad rings (remove, not weight 0):

d321 ms-be10 (10.0.6.209) sdc,

d294 ms-be7 (10.0.6.206) sda (which was replaced but we can't get linux to see it so whatever)

add weight to ms-be8 (10.0.6.207) to 100

add weight to ms-be10 (10.0.6.209) to 100

• Fri Dec 14

remove weight from ms-be5 (10.0.6.204) to 66

add weight to ms-be7 (10.0.6.206) to 100

• Mon Dec 17 (probably will slip)

remove weight from ms-be5 to 33

put in new ms-be3 to 33

• Mon Dec 17 -- dates from here on need to be recalculated

remove weight from ms-be5 to 0

add weight to ms-be3 to 66

• Wed Dec 19

power off ms-be5

remove weight from ms-be11 to 66

add weight to ms-be3 to 100

• Fri Dec 21

remove weight from ms-be11 to 33

put in new ms-be5 at 33

• Mon Dec 24

remove weight from ms-be11 to 0

add weight to ms-be5 at 66

• Wed Dec 26

remove weight from ms-be12 to 66

power off ms-be11

add weight to ms-be5 at 100

• Fri Dec 28

remove weight from ms-be12 to 33

put in new ms-be11 at 33

• Mon Dec 31

remove weight from ms-be12 to 0

add weight to ms-be11 at 66

• Wed Jan 2

remove weight from me-be9 to 66

power off ms-be12

add weight to ms-be11 at 100

• Fri Jan 4

remove weight from ms-be9 to 33

put in new ms-be12 to 33

• Mon Jan 7

remove weight from ms-be9 to 0

add weight to ms-be12 to 66

• Wed Jan 9

remove weight from ms-be2 to 66

power off ms-be9

add weight to ms-be12 to 100

• Fri Jan 11

remove weight from ms-be2 to 33

put in new ms-be9 to 33

• Mon Jan 14

remove weight from ms-be2 to 0

add weight to ms-be9 to 66

• Wed Jan 16

remove weight from ms-be1 to 66

power off ms-be2

add weight to ms-be9 to 100

• Fri Jan 18

remove weight from ms-be1 to 33

put in new ms-be2 to 33

• Mon Jan 21

remove weight from ms-be1 to 0

add weight to ms-be2 to 66

• Wed Jan 23

remove weight from ms-be4 to 66

power off ms-be1

add weight to ms-be2 to 100

• Fri Jan 25

remove weight from ms-be4 to 33

put in new ms-be1 to 33

• Mon Jan 28

remove weight from ms-be4 to 0

add weight to ms-be1 to 66

• Wed Jan 31

power off ms-be4

add weight to ms-be1 to 100

put in new ms-be4 at 33

• Fri Feb 2

add weight to ms-be4 to 66

• Mon Feb 5

add weight to ms-be4 to 100

This schedule will be adjusted (shorter or longer) as we see how the cluster behaves.

Note: all hosts get ssds

Logistics

• What is our deadline for returning the C2100s?
• Do we have enough ssds to put in all remaining 720xds? If not let's get them (and what's the turnaround time for those?) YES we do.