6TB storage expansion
Contents
Introduction
56 instead of 50TB are now available for the bioinformatics cluster. The tasks required to realize this extra space are:
- rescan multipath, details: multipath WWID; 36000d3100051f800000000000000016e (on mpathd)
Tools
listing
multipath -l
will give a useful if not very understandable list of characteristics. Most important, the device: dm-6 is noted, i.e.
mpathd (36000d3100051f800000000000000016e) dm-6 COMPELNT,Compellent Vol size=50T features='1 queue_if_no_path' hwhandler='0' wp=rw `-+- policy='round-robin 0' prio=0 status=active |- 6:0:0:1 sdd 8:48 active undef running |- 7:0:1:1 sde 8:64 active undef running |- 6:0:1:1 sdg 8:96 active undef running `- 7:0:2:1 sdf 8:80 active undef running
configuration
/etc/multipath.conf
Doesn't really have much. The only actives are:
defaults {
user_friendly_names yes
}
blacklist {
wwid 3600605b006c5f11019eadc840e589ae9
wwid 3600605b006a1af8019ee9a2d1036bfa7
wwid 3600605b006a1af801b46c4773d54b7d4
}
interpreter
There is also a interpreter available to query status and other things. To fire it up, type
multipathd -k
The
show maps topology
command in this interpeter will give you the same output as the "multipath -l" command. It mayy be necessary to find out more about the tool. For example, this command
paths count
gives 4 as the answer, which probably refers to sdd, sde, sdg, sdf mentioned in the above output. Indeed the
show devices
gives these four as the output. Unfortunately, there is nothign about rescanning here. I think this tool is at a higher-level than what is required for the rescanning, which seems to be a lower level jobs (multipath's understanding of "devices" gives this away somewhat).
Indeed this tool will be useful for resizing, but unfortunately not for rescanning.
SCSI Considerations
There is a way to see the SCSI devices:
cat /proc/scsi/scsi
Output:
Attached devices: Host: scsi8 Channel: 00 Id: 08 Lun: 00 Vendor: AIC CORP Model: SAS 6G Expander Rev: 0b05 Type: Enclosure ANSI SCSI revision: 05 Host: scsi8 Channel: 00 Id: 09 Lun: 00 Vendor: AIC CORP Model: SAS 6G Expander Rev: 0b05 Type: Enclosure ANSI SCSI revision: 05 Host: scsi8 Channel: 00 Id: 10 Lun: 00 Vendor: AIC CORP Model: SAS 6G Expander Rev: 0b05 Type: Enclosure ANSI SCSI revision: 05 Host: scsi8 Channel: 00 Id: 11 Lun: 00 Vendor: AIC CORP Model: SAS 6G Expander Rev: 0b05 Type: Enclosure ANSI SCSI revision: 05 Host: scsi8 Channel: 02 Id: 00 Lun: 00 Vendor: LSI Model: MR9286-8e Rev: 3.29 Type: Direct-Access ANSI SCSI revision: 05 Host: scsi9 Channel: 02 Id: 00 Lun: 00 Vendor: LSI Model: MR9261-8i Rev: 2.13 Type: Direct-Access ANSI SCSI revision: 05 Host: scsi9 Channel: 02 Id: 01 Lun: 00 Vendor: LSI Model: MR9261-8i Rev: 2.13 Type: Direct-Access ANSI SCSI revision: 05 Host: scsi6 Channel: 00 Id: 00 Lun: 01 Vendor: COMPELNT Model: Compellent Vol Rev: 0602 Type: Direct-Access ANSI SCSI revision: 05 Host: scsi7 Channel: 00 Id: 01 Lun: 01 Vendor: COMPELNT Model: Compellent Vol Rev: 0602 Type: Direct-Access ANSI SCSI revision: 05 Host: scsi7 Channel: 00 Id: 02 Lun: 01 Vendor: COMPELNT Model: Compellent Vol Rev: 0602 Type: Direct-Access ANSI SCSI revision: 05 Host: scsi6 Channel: 00 Id: 01 Lun: 01 Vendor: COMPELNT Model: Compellent Vol Rev: 0602 Type: Direct-Access ANSI SCSI revision: 05
There is a relationship to the "multipath -l" output in terms of the vendor and model.
There is also a rescan tool for SCSI
rescan-scsi-bus.sh
Out put is a bit voluminous, but here goes:
Scanning SCSI subsystem for new devices
Scanning host 0 for SCSI target IDs 0 1 2 3 4 5 6 7, all LUNs
Scanning host 1 for SCSI target IDs 0 1 2 3 4 5 6 7, all LUNs
Scanning host 2 for SCSI target IDs 0 1 2 3 4 5 6 7, all LUNs
Scanning host 3 for SCSI target IDs 0 1 2 3 4 5 6 7, all LUNs
Scanning host 4 for SCSI target IDs 0 1 2 3 4 5 6 7, all LUNs
Scanning host 5 for SCSI target IDs 0 1 2 3 4 5 6 7, all LUNs
Scanning host 6 for all SCSI target IDs, all LUNs
sg7 changed: device 6 0 0 1 ...
from:Direct-Access : 01
to: Direct-Access Model: Compellent Vol Rev: 0602
Type: Direct-Access ANSI SCSI revision: 05
sg10 changed: device 6 0 1 1 ...
from:Direct-Access : 01
to: Direct-Access Model: Compellent Vol Rev: 0602
Type: Direct-Access ANSI SCSI revision: 05
Scanning host 7 for all SCSI target IDs, all LUNs
sg8 changed: device 7 0 1 1 ...
from:Direct-Access : 01
to: Direct-Access Model: Compellent Vol Rev: 0602
Type: Direct-Access ANSI SCSI revision: 05
sg9 changed: device 7 0 2 1 ...
from:Direct-Access : 01
to: Direct-Access Model: Compellent Vol Rev: 0602
Type: Direct-Access ANSI SCSI revision: 05
Scanning host 8 for SCSI target IDs 0 1 2 3 4 5 6 7, all LUNs
sg2 changed: device 8 0 10 0 ...
from:Enclosure : 00
to: Enclosure Model: SAS 6G Expander Rev: 0b05
Type: Enclosure ANSI SCSI revision: 05
sg3 changed: device 8 0 11 0 ...
from:Enclosure : 00
to: Enclosure Model: SAS 6G Expander Rev: 0b05
Type: Enclosure ANSI SCSI revision: 05
sg0 changed: device 8 0 8 0 ...
from:Enclosure : 00
to: Enclosure Model: SAS 6G Expander Rev: 0b05
Type: Enclosure ANSI SCSI revision: 05
sg1 changed: device 8 0 9 0 ...
from:Enclosure : 00
to: Enclosure Model: SAS 6G Expander Rev: 0b05
Type: Enclosure ANSI SCSI revision: 05
sg4 changed: device 8 2 0 0 ...
from:Direct-Access : 00
to: Direct-Access Model: MR9286-8e Rev: 3.29
Type: Direct-Access ANSI SCSI revision: 05
Scanning host 9 for SCSI target IDs 0 1 2 3 4 5 6 7, all LUNs
sg5 changed: device 9 2 0 0 ...
from:Direct-Access : 00
to: Direct-Access Model: MR9261-8i Rev: 2.13
Type: Direct-Access ANSI SCSI revision: 05
sg6 changed: device 9 2 1 0 ...
from:Direct-Access : 00
to: Direct-Access Model: MR9261-8i Rev: 2.13
Type: Direct-Access ANSI SCSI revision: 05
0 new or changed device(s) found.
0 remapped or resized device(s) found.
0 device(s) removed.
Despite some mentions of changed devices, the final summary is of no changes.
There is a pattern to the device numbered 6 however, which is always a Compellent volume.
Procedure
The rescan operation is most definitely required. Both Red Hat and Ubuntu guides give the same method of doing this
# echo 1 > /sys/block/device_name/device/rescan
However, under /sys/block/dm-6, there is no device subdirectory, and, needless to say, no rescan file. Mind, this is for SCSI.
Other sites repeat the recipe ad nauseam. There is no mention about whether this can in fact be done live.
Breakthrough
Here is the raw recipe:
# echo 1 > /sys/block/device_name/device/rescan # multipathd -k'resize map mpatha'
The realisation is that the device name is not dm-6, and there could be more than one device. Namely, the sd{d,e,f,g} mentioned in the "multipath -l" command are the referred-to devices. This becomes clear when we see that each have a device/rescan file which is write-only (we noted above that dm-6 has no such directory nor file). Also, attempts to create such a path/rescan directory and file in /sys/block/dm-6 were fruitless. Therefore 1 wa echo'd to each of the four rescan files in the device subdirectory of /sys/block/sd{d,e,f,g}
This was accepted. Then came the resize command in the multipath interpreter. This returned a "fail". This is because mpatha is a placeholder (recipe was mute about this) and that it should be replaced with the target map in your own configuration. In the interpreter
show maps
will offer the name of the map whihc is marvin's case was mapthd. So, to resize, the command really is
multipathd -k'resize map mpathd'
This returned a positive message, and finally "multipath -l" gave the new size as 56T, hurrah.
The next thing that has to be done is extend the logical volumes and grow the XFS.
LVM
A quick pvscan will reveal that the /dev/mapper/mpathd device has no free space on it. So the extra 6TB has not been recognised? Well, it's because the PV needs to be resized. This can be done with:
pvresize /dev/mapper/mpathd
This will show how one PV was resized and 0 were not changed (unusually phrased message), and a quick pvscan will show that the PV now has 6TB extra space. Next up is the logical volume .. this must be extended into newly resized PV. This is done with the follwong command:
lvextend /dev/STORAGE/storage /dev/mapper/mpathd
XFS
xfs is very easy to resize and can famously be done while filesystem is already mounted.
xfs_growfs /dev/STORAGE/storage
This command returned extremely quickly and showed that the filesystem had been resize from 50T to 56T. Mission accomplished!
Summary of successful operation
echo 1 > /sys/block/sdd/device/rescan echo 1 > /sys/block/sde/device/rescan echo 1 > /sys/block/sdf/device/rescan echo 1 > /sys/block/sdg/device/rescan multipathd -k'resize map mpathd' pvresize /dev/mapper/mpathd lvextend /dev/STORAGE/storage /dev/mapper/mpathd xfs_growfs /dev/STORAGE/storage
