Sun Fire X4500 and X4540

Latest revision as of 04:32, 7 September 2012

• ms1, ms2: X4500
• ms3, ms4, ms5, ms6: X4540

The X4500 and X4540 are non-clustered storage servers from Sun, with 48 SATA disks. We use them for upload storage, and will be using them for external storage. The X4500 is the original model, with 2 dual-core CPUs, 16GB RAM, and marvell88sx(7d) SATA controllers. The X4540 has 2 quad-core CPUs, 32GB RAM, and LSI mpt(7d) SAS/SATA controllers. Neither of these cards support hardware RAID; instead, we use Solaris with ZFS.

Both systems have hot-swappable disks, and the systems are racked to allow them to be fully extended out on their rails while remaining powered on with network connectivity.

Disk layout diagrams:

[edit] Sun X4540

---------------------SunFireX4540-------Rear----------------------------
 3:    7:   11:   15:   19:   23:   27:   31:   35:   39:   43:   47:   
c0t3  c0t7  c1t3  c1t7  c2t3  c2t7  c3t3  c3t7  c4t3  c4t7  c5t3  c5t7

 2:    6:   10:   14:   18:   22:   26:   30:   34:   38:   42:   46:   
c0t2  c0t6  c1t2  c1t6  c2t2  c2t6  c3t2  c3t6  c4t2  c4t6  c5t2  c5t6  

 1:    5:    9:   13:   17:   21:   25:   29:   33:   37:   41:   45:   
c0t1  c0t5  c1t1  c1t5  c2t1  c2t5  c3t1  c3t5  c4t1  c4t5  c5t1  c5t5  
 b           b   
 0:    4:    8:   12:   16:   20:   24:   28:   32:   36:   40:   44:   
c0t0  c0t4  c1t0  c1t4  c2t0  c2t4  c3t0  c3t4  c4t0  c4t4  c5t0  c5t4  
 b           b   
-------*-----------*-SunFireX4540---*---Front----*---------*--------

[edit] Sun X4500

[edit] Solaris

---------------------SunFireX4500------Rear----------------------------

36:   37:   38:   39:   40:   41:   42:   43:   44:   45:   46:   47:   
c5t3  c5t7  c4t3  c4t7  c7t3  c7t7  c6t3  c6t7  c1t3  c1t7  c0t3  c0t7  

24:   25:   26:   27:   28:   29:   30:   31:   32:   33:   34:   35:   
c5t2  c5t6  c4t2  c4t6  c7t2  c7t6  c6t2  c6t6  c1t2  c1t6  c0t2  c0t6  

12:   13:   14:   15:   16:   17:   18:   19:   20:   21:   22:   23:   
c5t1  c5t5  c4t1  c4t5  c7t1  c7t5  c6t1  c6t5  c1t1  c1t5  c0t1  c0t5  

 0:    1:    2:    3:    4:    5:    6:    7:    8:    9:   10:   11:   
c5t0  c5t4  c4t0  c4t4  c7t0  c7t4  c6t0  c6t4  c1t0  c1t4  c0t0  c0t4  
 b     b       
-------*-----------*-SunFireX4500--*---Front-----*-----------*----------

[edit] Linux

---------------------SunFireX4500------Rear----------------------------

36:   37:   38:   39:   40:   41:   42:   43:   44:   45:   46:   47:   
c3t3  c3t7  c2t3  c2t7  c5t3  c5t7  c4t3  c4t7  c1t3  c1t7  c0t3  c0t7  

24:   25:   26:   27:   28:   29:   30:   31:   32:   33:   34:   35:   
c3t2  c3t6  c2t2  c2t6  c5t2  c5t6  c4t2  c4t6  c1t2  c1t6  c0t2  c0t6  

12:   13:   14:   15:   16:   17:   18:   19:   20:   21:   22:   23:   
c3t1  c3t5  c2t1  c2t5  c5t1  c5t5  c4t1  c4t5  c1t1  c1t5  c0t1  c0t5  

 0:    1:    2:    3:    4:    5:    6:    7:    8:    9:   10:   11:   
c3t0  c3t4  c2t0  c2t4  c5t0  c5t4  c4t0  c4t4  c1t0  c1t4  c0t0  c0t4  
 b     b       
-------*-----------*-SunFireX4500--*---Front-----*-----------*----------

• X4500: The OS drives are the two leftmost at the front (controller 5, disks 0 & 4), labelled 'b' in the diagram. No other drives show in the BIOS during post.
• X4540: There are four OS disks, labelled 'b' in the diagram.

The disk serial numbers are readable when the top cover is off!

It helps to disable the IDE/PATA controllers in the BIOS to get more consistent SCSI device naming!

The disks are (currently) configured as a RAID-Z pool in 9 striped sets of 5; total space is ~8TB (250GB disks) or ~16TB (500G disks). It's mounted at /export.

Note: under linux we had some trouble determining which disks were which. Some speculation is available on the talk page.

[edit] Solaris disk names under Linux

http://lists.lustre.org/pipermail/linux_hpc_swstack/2008-June/000036.html has a non-working way to use Solaris like disk naming under Linux.

Mark rewrote this using valid udev syntax. This is only valid for the X4500 (Thumper).

First, make sure the two IDE controllers are not interfering with SCSI ID naming. Disable the pata_amd driver: make a file /etc/modprobe.d/blacklist-pata:

blacklist pata_amd

Then, run:

# depmod -ae
# update-initramfs -u

Solaris device name symlinks can be made by udev. Put the following in /etc/udev/rules.d/99-thumper-disks.rules:

#
# /etc/udev/rules.d/99-thumper-disks.rules
# Written on 2009/08/15 by Mark Bergsma <mark@nedworks.org>
#

# Disk rules
SUBSYSTEM=="block", KERNEL=="sd*[a-z]", KERNELS=="*:0:0:0", PROGRAM="/etc/udev/scripts/solaris-name.sh %b", ENV{SOLARIS_NAME}="$result", SYMLINK+="disk/by-cntrl/$env{SOLARIS_NAME}"

# Partition rules
SUBSYSTEM=="block", KERNEL=="sd*[a-z][0-9]*", KERNELS=="*:0:0:0", PROGRAM="/etc/udev/scripts/solaris-name.sh %b %n", ENV{SOLARIS_NAME}="$result", SYMLINK+="disk/by-cntrl/$env{SOLARIS_NAME}"

And this (executable!) helper script in /etc/udev/scripts/solaris-name.sh:

#!/bin/bash

seq=$(echo $1 | cut -d':' -f 1)
controller=$(($seq / 8))
disk=$((seq % 8))

if [ -n "$2" ]; then
	echo "c${controller}t${disk}d0s$(($2 - 1))"
else
	echo "c${controller}t${disk}d0"
fi

[edit] Replace a disk

[edit] X4540

Look at zpool status to find the failed disk:

       NAME          STATE     READ WRITE CKSUM
       export        DEGRADED     0     0     0
         raidz1      ONLINE       0     0     0
           c5t0d0    ONLINE       0     0     0
           c5t1d0    ONLINE       0     0     0
           c5t2d0    ONLINE       0     0     0
           c5t3d0    ONLINE       0     0     0
           c5t4d0    ONLINE       0     0     0
         raidz1      DEGRADED     0     0     0
           spare     DEGRADED     0     0     0
             c5t5d0  REMOVED      0     0     0
             c1t7d0  ONLINE       0     0     0
           c5t6d0    ONLINE       0     0     0
           c5t7d0    ONLINE       0     0     0
           c2t0d0    ONLINE       0     0     0
           c2t1d0    ONLINE       0     0     0

replace the physical disk, then run:

zpool replace export c5t5d0

(assuming c5t5d0 is the failed disk.) The disk will be resilvered, then the spare will be removed from the pool.

[edit] X4500

To replace a disk, it must first be removed from the zpool. The easiest way to do this is to replace it with the hot spare. For example, if c7t6d0 is being removed, and c0t2d0 is the hot spare:

# zpool replace export c7t6d0 c0t2d0

Wait for the resilver to complete (it'll take a few hours; see 'zpool status' for progress). Then remove the disk from the pool:

# zpool detach export c7t6d0

Identify the disk in cfgadm and unconfigure it:

# cfgadm -l | grep c7t6d0
sata5/6::dsk/c7t6d0            disk         connected    configured   ok
# cfgadm -c unconfigure sata5/6
Unconfigure the device at: /devices/pci@2,0/pci1022,7458@8/pci11ab,11ab@1:6
This operation will suspend activity on the SATA device
Continue (yes/no)? yes

The blue LED on the disk should illuminate. Replace the disk, re-configure it, and add the new disk to the zpool as a hot spare:

# cfgadm -c configure sata5/6
# zpool add export spare c7t6d0

[edit] OS setup

Press Ctrl-N during bootup for network boot.

[edit] Linux

Under Lucid, the install on the X4500s fail because grub doesn't understand device name /dev/sdac. On /dev/sdy it works.

Just go into a shell, chroot to /target, and use:

# grub-install '(hd1)'

[edit] Solaris

This is how the OS (Solaris) is set up on the image servers (ms1/ms5).

Solaris 10 is installed. Docs are at http://docs.sun.com/. All systems except ms1 are using ZFS root.

Disks c5t0d0 and c5t4d0 (X4500) or c0t0d0 and c1t0d0 (X4540) are mirrored for the root ZFS pool and swap.

For software not included in Solaris, we use the Toolserver software repository. This is in SVN at /trunk/tools/ts-specs. Get by:

 svn co https://svn.toolserver.org/svnroot/toolserver/trunk/ts-specs

These are RPM spec files that can be built into Solaris packages with pkgtool.

$ pkgtool -v build-only --download TSwhatever.spec
$ su
# pkgadd -d $HOME/packages TSwhatever

Most software from this repository installs into /opt/ts. PHP is in /opt/php for builds before July 2009. (Config file: /etc/opt/php/php.ini) For builds later than that it is in /opt/ts/php.

Service management is done with the Solaris SMF (Service Management Framework) (Reference: Solaris Administration Guide)

List services:

# svcs -a

Find out why a service is in 'maintenance' state instead of 'online':

# svcs -vx <name>

Disable a service:

# svcadm disable <name>

Enable a service:

# svcadm enable <name>

Disable a service but leave it enabled on the next boot:

# svcadm disable -t <name>

Enable a service but leave it disabled on the next boot:

# svcadm enable -t <name>

Service name can be abbreviated, e.g. 'lighttpd' instead of 'svc:/network/lighttpd:lighttpd'.

Lighttpd is the same as amane. Config: /etc/lighttpd/lighttpd.conf. It's built from TSlighttpd in the Toolserver repo.

For OS upgrades and patching, Live Upgrade should be used. This creates copies of the running OS. List current LU boot environments:

root@ms1:~# lustatus
Boot Environment           Is       Active Active    Can    Copy      
Name                       Complete Now    On Reboot Delete Status    
-------------------------- -------- ------ --------- ------ ----------
s10a                       yes      yes    yes       no     -  

Create a new boot environment as a copy of the current one:

# lucreate -n s10b

Mount the new environment:

# lumount s10b

List patches:

# /opt/ts/bin/pca -R /.alt.s10b

Apply patches to s10b and reboot:

# /opt/ts/bin/pca -R /.alt.s10b -ai
# luumount s10b
# luactivate s10b
# init 6

[edit] Lights Out Management

The Sun X4500 and X4540 uses iLOM.

The standard administrative account is root and the default password is changeme.

[edit] Common actions

[edit] Changing the root password

set /SP/users/root password=password

[edit] Serial console

start /SP/console

[edit] Power cycle

Reset:

reset /SYS

Power Off:

stop /SYS

Power On:

start /SYS

End the session with Esc-(

[edit] Setting an IP address for the network port of LOM

Login to the LOM over serial and run the following:

cd /SP/network 
set pendingipaddress=ipaddress
set pendingipnetmask=255.255.0.0
set pendingipgateway=10.1.0.1
set pendingipdiscovery=static
set commitpending=true

Once that is done, you should be able to connect over the management IP if it is plugged into the management network.

[edit] External links

• Sun X4500 documentation
• Sun Integrated Lights Out Manager 2.0 User's Guide