Swift/Deploy Plan - R720xds in tampa

From Wikitech
< Swift(Difference between revisions)
Jump to: navigation, search
(→‎Proposed schedule)
Line 9: Line 9:
  
 
===Proposed schedule===
 
===Proposed schedule===
 +
Bah, the last proposed schedule died a horrible death; it was taking 5 days for one complete run of the object replicators.  After some back of the napkin calculations, here's a new schedule (old one is hidden in html comments).  I'll be monitoring to see if the object replication runs actually finish in two days when we move this much data around; if they do, we'll try the below:
  
 +
* Mon Dec 10 -- done
 +
:take out the two bad rings (remove, not weight 0):
 +
:  d321  ms-be10 (10.0.6.209) sdc,
 +
:  d294  ms-be7 (10.0.6.206) sda (which was replaced but we can't get linux to see it so whatever)
 +
:add weight to ms-be8 (10.0.6.207) to 100
 +
:add weight to ms-be10 (10.0.6.209) to 100
 +
 +
* Wed Dec 12
 +
:remove weight from ms-be5 (10.0.6.204) to 66
 +
:add weight to ms-be7 to 100
 +
 +
* Fri Dec 14
 +
:remove weight from ms-be5 to 33
 +
:put in new ms-be3 to 33
 +
 +
* Mon Dec 17
 +
:remove weight from ms-be5 to 0
 +
:add weight to ms-be3 to 66
 +
 +
* Wed Dec 19
 +
:power off ms-be5
 +
:remove weight from ms-be11 to 66
 +
:add weight to ms-be3 to 100
 +
 +
* Fri Dec 21
 +
:remove weight from ms-be11 to 33
 +
:put in new ms-be5 at 33
 +
 +
* Mon Dec 24
 +
:remove weight from ms-be11 to 0
 +
:add weight to ms-be5 at 66
 +
 +
* Wed Dec 26
 +
:remove weight from ms-be12 to 66
 +
:power off ms-be11
 +
:add weight to ms-be5 at 100
 +
 +
* Fri Dec 28
 +
:remove weight from ms-be12 to 33
 +
:put in new ms-be11 at 33
 +
 +
* Mon Dec 31
 +
:remove weight from ms-be12 to 0
 +
:add weight to ms-be11 at 66
 +
 +
* Wed Jan 2
 +
:remove weight from me-be9 to 66
 +
:power off ms-be12
 +
:add weight to ms-be11 at 100
 +
 +
* Fri Jan 4
 +
:remove weight from ms-be9 to 33
 +
:put in new ms-be12 to 33
 +
 +
* Mon Jan 7
 +
:remove weight from ms-be9 to 0
 +
:add weight to ms-be12 to 66
 +
 +
* Wed Jan 9
 +
:remove weight from ms-be2 to 66
 +
:power off ms-be9
 +
:add weight to ms-be12 to 100
 +
 +
* Fri Jan 11
 +
:remove weight from ms-be2 to 33
 +
:put in new ms-be9 to 33
 +
 +
* Mon Jan 14
 +
:remove weight from ms-be2 to 0
 +
:add weight to ms-be9 to 66
 +
 +
* Wed Jan 16
 +
:remove weight from ms-be1 to 66
 +
:power off ms-be2
 +
:add weight to ms-be9 to 100
 +
 +
* Fri Jan 18
 +
:remove weight from ms-be1 to 33
 +
:put in new ms-be2 to 33
 +
 +
* Mon Jan 21
 +
:remove weight from ms-be1 to 0
 +
:add weight to ms-be2 to 66
 +
 +
* Wed Jan 23
 +
:remove weight from ms-be4 to 66
 +
:power off ms-be1
 +
:add weight to ms-be2 to 100
 +
 +
* Fri Jan 25
 +
:remove weight from ms-be4 to 33
 +
:put in new ms-be1 to 33
 +
 +
* Mon Jan 28
 +
:remove weight from ms-be4 to 0
 +
:add weight to ms-be1 to 66
 +
 +
* Wed Jan 31
 +
:power off ms-be4
 +
:add weight to ms-be1 to 100
 +
:put in new ms-be4 at 33
 +
 +
* Fri Feb 2
 +
:add weight to ms-be4 to 66
 +
 +
* Mon Feb 5
 +
:add weight to ms-be4 to 100
 +
 +
<!--
 
Pushing our luck a little bit, we can go from 33 weight for the object rings to 66 in 3 days and from 66 to 100 in another 4.  This is a compressed schedule, it really doesn't let network traffic to the new hosts settle down completely, but it will let us get the new boxes in and the old ones shipped.
 
Pushing our luck a little bit, we can go from 33 weight for the object rings to 66 in 3 days and from 66 to 100 in another 4.  This is a compressed schedule, it really doesn't let network traffic to the new hosts settle down completely, but it will let us get the new boxes in and the old ones shipped.
  
Line 25: Line 135:
 
*me be-2 on Dec 17
 
*me be-2 on Dec 17
 
*ms ms-be4 on Dec 19
 
*ms ms-be4 on Dec 19
 +
 +
-->
  
 
This schedule will be adjusted (shorter or longer) as we see how the cluster behaves.
 
This schedule will be adjusted (shorter or longer) as we see how the cluster behaves.

Revision as of 16:58, 10 December 2012

Draft, edit at will

Current status

  • Currently replaced: ms-be6,7,8,10
  • Host which has fallen over a few times recently: ms-be3
  • Hosts with disk issues (so they should be replaced next): ms-be5,11,12
  • Other hosts with ssds: ms-be9
  • The rest: ms-be1,2,4

Proposed schedule

Bah, the last proposed schedule died a horrible death; it was taking 5 days for one complete run of the object replicators. After some back of the napkin calculations, here's a new schedule (old one is hidden in html comments). I'll be monitoring to see if the object replication runs actually finish in two days when we move this much data around; if they do, we'll try the below:

  • Mon Dec 10 -- done
take out the two bad rings (remove, not weight 0):
d321 ms-be10 (10.0.6.209) sdc,
d294 ms-be7 (10.0.6.206) sda (which was replaced but we can't get linux to see it so whatever)
add weight to ms-be8 (10.0.6.207) to 100
add weight to ms-be10 (10.0.6.209) to 100
  • Wed Dec 12
remove weight from ms-be5 (10.0.6.204) to 66
add weight to ms-be7 to 100
  • Fri Dec 14
remove weight from ms-be5 to 33
put in new ms-be3 to 33
  • Mon Dec 17
remove weight from ms-be5 to 0
add weight to ms-be3 to 66
  • Wed Dec 19
power off ms-be5
remove weight from ms-be11 to 66
add weight to ms-be3 to 100
  • Fri Dec 21
remove weight from ms-be11 to 33
put in new ms-be5 at 33
  • Mon Dec 24
remove weight from ms-be11 to 0
add weight to ms-be5 at 66
  • Wed Dec 26
remove weight from ms-be12 to 66
power off ms-be11
add weight to ms-be5 at 100
  • Fri Dec 28
remove weight from ms-be12 to 33
put in new ms-be11 at 33
  • Mon Dec 31
remove weight from ms-be12 to 0
add weight to ms-be11 at 66
  • Wed Jan 2
remove weight from me-be9 to 66
power off ms-be12
add weight to ms-be11 at 100
  • Fri Jan 4
remove weight from ms-be9 to 33
put in new ms-be12 to 33
  • Mon Jan 7
remove weight from ms-be9 to 0
add weight to ms-be12 to 66
  • Wed Jan 9
remove weight from ms-be2 to 66
power off ms-be9
add weight to ms-be12 to 100
  • Fri Jan 11
remove weight from ms-be2 to 33
put in new ms-be9 to 33
  • Mon Jan 14
remove weight from ms-be2 to 0
add weight to ms-be9 to 66
  • Wed Jan 16
remove weight from ms-be1 to 66
power off ms-be2
add weight to ms-be9 to 100
  • Fri Jan 18
remove weight from ms-be1 to 33
put in new ms-be2 to 33
  • Mon Jan 21
remove weight from ms-be1 to 0
add weight to ms-be2 to 66
  • Wed Jan 23
remove weight from ms-be4 to 66
power off ms-be1
add weight to ms-be2 to 100
  • Fri Jan 25
remove weight from ms-be4 to 33
put in new ms-be1 to 33
  • Mon Jan 28
remove weight from ms-be4 to 0
add weight to ms-be1 to 66
  • Wed Jan 31
power off ms-be4
add weight to ms-be1 to 100
put in new ms-be4 at 33
  • Fri Feb 2
add weight to ms-be4 to 66
  • Mon Feb 5
add weight to ms-be4 to 100


This schedule will be adjusted (shorter or longer) as we see how the cluster behaves.

Note: all hosts get ssds

Logistics

  • What is our deadline for returning the C2100s?
  • Do we have enough ssds to put in all remaining 720xds? If not let's get them (and what's the turnaround time for those?) YES we do.
Retrieved from "http://wikitech-old.wikimedia.org/index.php?title=Swift/Deploy_Plan_-_R720xds_in_tampa&oldid=54410"
Personal tools
Namespaces

Variants
Actions
Navigation
Ops documentation
Wiki
Toolbox