6TB storage expansion

Introduction

56 instead of 50TB are now available for the bioinformatics cluster. The tasks required to realize this extra space are:

• rescan multipath, details: multipath WWID; 36000d3100051f800000000000000016e (on mpathd)

Tools

listing

multipath -l

will give a useful if not very understandable list of characteristics. Most important, the device: dm-6 is noted, i.e.

mpathd (36000d3100051f800000000000000016e) dm-6 COMPELNT,Compellent Vol
size=50T features='1 queue_if_no_path' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=0 status=active
  |- 6:0:0:1 sdd 8:48 active undef running
  |- 7:0:1:1 sde 8:64 active undef running
  |- 6:0:1:1 sdg 8:96 active undef running
  `- 7:0:2:1 sdf 8:80 active undef running

configuration

/etc/multipath.conf

Doesn't really have much. The only actives are:

defaults {
   user_friendly_names yes
}

blacklist {
    wwid 3600605b006c5f11019eadc840e589ae9
    wwid 3600605b006a1af8019ee9a2d1036bfa7
    wwid 3600605b006a1af801b46c4773d54b7d4
}

interpreter

There is also a interpreter available to query status and other things. To fire it up, type

multipathd -k

The

show maps topology

command in this interpeter will give you the same output as the "multipath -l" command. It mayy be necessary to find out more about the tool. For example, this command

paths count

gives 4 as the answer, which probably refers to sdd, sde, sdg, sdf mentioned in the above output. Indeed the

show devices

gives these four as the output. Unfortunately, there is nothign about rescanning here. I think this tool is at a higher-level than what is required for the rescanning, which seems to be a lower level jobs (multipath's understanding of "devices" gives this away somewhat).

Indeed this tool will be useful for resizing, but unfortunately not for rescanning.

SCSI Considerations

There is a way to see the SCSI devices:

cat /proc/scsi/scsi

Output:

Attached devices:
Host: scsi8 Channel: 00 Id: 08 Lun: 00
  Vendor: AIC CORP Model: SAS 6G Expander  Rev: 0b05
  Type:   Enclosure                        ANSI  SCSI revision: 05
Host: scsi8 Channel: 00 Id: 09 Lun: 00
  Vendor: AIC CORP Model: SAS 6G Expander  Rev: 0b05
  Type:   Enclosure                        ANSI  SCSI revision: 05
Host: scsi8 Channel: 00 Id: 10 Lun: 00
  Vendor: AIC CORP Model: SAS 6G Expander  Rev: 0b05
  Type:   Enclosure                        ANSI  SCSI revision: 05
Host: scsi8 Channel: 00 Id: 11 Lun: 00
  Vendor: AIC CORP Model: SAS 6G Expander  Rev: 0b05
  Type:   Enclosure                        ANSI  SCSI revision: 05
Host: scsi8 Channel: 02 Id: 00 Lun: 00
  Vendor: LSI      Model: MR9286-8e        Rev: 3.29
  Type:   Direct-Access                    ANSI  SCSI revision: 05
Host: scsi9 Channel: 02 Id: 00 Lun: 00
  Vendor: LSI      Model: MR9261-8i        Rev: 2.13
  Type:   Direct-Access                    ANSI  SCSI revision: 05
Host: scsi9 Channel: 02 Id: 01 Lun: 00
  Vendor: LSI      Model: MR9261-8i        Rev: 2.13
  Type:   Direct-Access                    ANSI  SCSI revision: 05
Host: scsi6 Channel: 00 Id: 00 Lun: 01
  Vendor: COMPELNT Model: Compellent Vol   Rev: 0602
  Type:   Direct-Access                    ANSI  SCSI revision: 05
Host: scsi7 Channel: 00 Id: 01 Lun: 01
  Vendor: COMPELNT Model: Compellent Vol   Rev: 0602
  Type:   Direct-Access                    ANSI  SCSI revision: 05
Host: scsi7 Channel: 00 Id: 02 Lun: 01
  Vendor: COMPELNT Model: Compellent Vol   Rev: 0602
  Type:   Direct-Access                    ANSI  SCSI revision: 05
Host: scsi6 Channel: 00 Id: 01 Lun: 01
  Vendor: COMPELNT Model: Compellent Vol   Rev: 0602
  Type:   Direct-Access                    ANSI  SCSI revision: 05

There is a relationship to the "multipath -l" output in terms of the vendor and model.

There is also a rescan tool for SCSI

rescan-scsi-bus.sh

Out put is a bit voluminous, but here goes:

Scanning SCSI subsystem for new devices
Scanning host 0 for  SCSI target IDs  0 1 2 3 4 5 6 7, all LUNs
Scanning host 1 for  SCSI target IDs  0 1 2 3 4 5 6 7, all LUNs
Scanning host 2 for  SCSI target IDs  0 1 2 3 4 5 6 7, all LUNs
Scanning host 3 for  SCSI target IDs  0 1 2 3 4 5 6 7, all LUNs
Scanning host 4 for  SCSI target IDs  0 1 2 3 4 5 6 7, all LUNs
Scanning host 5 for  SCSI target IDs  0 1 2 3 4 5 6 7, all LUNs
Scanning host 6 for  all SCSI target IDs, all LUNs

sg7 changed:  device 6 0 0 1 ...
from:Direct-Access                     : 01
to: Direct-Access      Model: Compellent Vol   Rev: 0602
      Type:   Direct-Access                    ANSI SCSI revision: 05

sg10 changed: device 6 0 1 1 ...
from:Direct-Access                     : 01
to: Direct-Access      Model: Compellent Vol   Rev: 0602
      Type:   Direct-Access                    ANSI SCSI revision: 05

Scanning host 7 for  all SCSI target IDs, all LUNs
sg8 changed:  device 7 0 1 1 ...
from:Direct-Access                     : 01
to: Direct-Access      Model: Compellent Vol   Rev: 0602
      Type:   Direct-Access                    ANSI SCSI revision: 05

sg9 changed:  device 7 0 2 1 ...
from:Direct-Access                     : 01
to: Direct-Access      Model: Compellent Vol   Rev: 0602
      Type:   Direct-Access                    ANSI SCSI revision: 05

Scanning host 8 for  SCSI target IDs  0 1 2 3 4 5 6 7, all LUNs
sg2 changed:  device 8 0 10 0 ...
from:Enclosure                         : 00
to: Enclosure          Model: SAS 6G Expander  Rev: 0b05
      Type:   Enclosure                        ANSI SCSI revision: 05

sg3 changed:  device 8 0 11 0 ...
from:Enclosure                         : 00
to: Enclosure          Model: SAS 6G Expander  Rev: 0b05
      Type:   Enclosure                        ANSI SCSI revision: 05

sg0 changed:  device 8 0 8 0 ...
from:Enclosure                         : 00
to: Enclosure          Model: SAS 6G Expander  Rev: 0b05
      Type:   Enclosure                        ANSI SCSI revision: 05

sg1 changed:  device 8 0 9 0 ...
from:Enclosure                         : 00
to: Enclosure          Model: SAS 6G Expander  Rev: 0b05
      Type:   Enclosure                        ANSI SCSI revision: 05

sg4 changed:  device 8 2 0 0 ...
from:Direct-Access                     : 00
to: Direct-Access      Model: MR9286-8e        Rev: 3.29
      Type:   Direct-Access                    ANSI SCSI revision: 05

Scanning host 9 for  SCSI target IDs  0 1 2 3 4 5 6 7, all LUNs
sg5 changed:  device 9 2 0 0 ...
from:Direct-Access                     : 00
to: Direct-Access      Model: MR9261-8i        Rev: 2.13
      Type:   Direct-Access                    ANSI SCSI revision: 05

sg6 changed:  device 9 2 1 0 ...
from:Direct-Access                     : 00
to: Direct-Access      Model: MR9261-8i        Rev: 2.13
      Type:   Direct-Access                    ANSI SCSI revision: 05

0 new or changed device(s) found.
0 remapped or resized device(s) found.
0 device(s) removed.

Despite some mentions of changed devices, the final summary is of no changes.

There is a pattern to the device numbered 6 however, which is always a Compellent volume.

Procedure

The rescan operation is most definitely required. Both Red Hat and Ubuntu guides give the same method of doing this

# echo 1 > /sys/block/device_name/device/rescan

However, under /sys/block/dm-6, there is no device subdirectory, and, needless to say, no rescan file. Mind, this is for SCSI.

Other sites repeat the recipe ad nauseam. There is no mention about whether this can in fact be done live.

Breakthrough

Here is the raw recipe:

# echo 1 > /sys/block/<device_name>/device/rescan 
# multipathd -k'resize map mpatha'

The realisation is that the device name is not dm-6, and there could be more than one device. Namely, the sd{d,e,f,g} mentioned in the "multipath -l" command are the referred-to devices. This becomes clear when we see that each have a device/rescan file which is write-only (we noted above that dm-6 has no such directory nor file). Also, attempts to create such a path/rescan directory and file in /sys/block/dm-6 were fruitless. Therefore 1 wa echo'd to each of the four rescan files in the device subdirectory of /sys/block/sd{d,e,f,g}

This was accepted. Then came the resize command in the multipath interpreter. This returned a "fail". This is because mpatha is a placeholder (recipe was mute about this) and that it should be replaced with the target map in your own configuration. In the interpreter

show maps

will offer the name of the map whihc is marvin's case was mapthd. So, to resize, the command really is

multipathd -k'resize map mpathd'

This returned a positive message, and finally "multipath -l" gave the new size as 56T, hurrah.

The next thing that has to be done is extend the logical volumes and grow the XFS.

LVM

A quick pvscan will reveal that the /dev/mapper/mpathd device has no free space on it. So the extra 6TB has not been recognised? Well, it's because the PV needs to be resized. This can be done with:

pvresize /dev/mapper/mpathd

This will show how one PV was resized and 0 were not changed (unusually phrased message), and a quick pvscan will show that the PV now has 6TB extra space. Next up is the logical volume .. this must be extended into newly resized PV. This is done with the follwong command:

lvextend /dev/STORAGE/storage /dev/mapper/mpathd

XFS

xfs is very easy to resize and can famously be done while filesystem is already mounted.

xfs_growfs /dev/STORAGE/storage

This command returned extremely quickly and showed that the filesystem had been resize from 50T to 56T. Mission accomplished!

Summary of successful operation

echo 1 > /sys/block/sdd/device/rescan 
echo 1 > /sys/block/sde/device/rescan
echo 1 > /sys/block/sdf/device/rescan 
echo 1 > /sys/block/sdg/device/rescan 
multipathd -k'resize map mpathd'
pvresize /dev/mapper/mpathd
lvextend /dev/STORAGE/storage /dev/mapper/mpathd
xfs_growfs /dev/STORAGE/storage

Links

Both of these are almost the same, but clearly the RedHat was hte preferred reference.

• Red Hat DM-Multipath guide
• ubuntu troubleshooting guide to DM-Multipath

This turned out to be not so relevant, included here for interest's sake:

• Kristian Reese's LUN solution