Backups

Introduction

It is very important to note that backups are usually only 7 days old.

In the case of accidental deletions, you must inform immediately as it could take several days to load up the snapshot, consider that it is many TB in size.

What it entails

• A file system has to be created from the snapshot.
• this is loaded onto the SAN
• The SCSI bus needs to be rescanned
• this snapshot-rendered-to-filesystem must then be mounted.

to restore files from back up

NOTE: This has not been done yet, but this is the theory.

Central IT present a SAN snapshot for the nearest time point.

You rescan for disks and mount it on a different mount point. If its lvm (which it is!!), you may need to change the disk ID on the snapshot.

https://blog.purestorage.com/cloning-lvm-volumes-using-pure-flashrecover-snapshots/

PS your backup retension is currently

a daily snapshot @00:01 , retained for a week and a monthly snapshot on first sunday of the month @04:00, retained for 14 weeks

scanning SCSI

root@marvin:/sys/class/scsi_host > rescan-scsi-bus.sh -a
Scanning SCSI subsystem for new devices
Scanning host 0 for  SCSI target IDs  0 1 2 3 4 5 6 7, all LUNs
Scanning host 1 for  SCSI target IDs  0 1 2 3 4 5 6 7, all LUNs
Scanning host 2 for  SCSI target IDs  0 1 2 3 4 5 6 7, all LUNs
Scanning host 3 for  SCSI target IDs  0 1 2 3 4 5 6 7, all LUNs
Scanning host 4 for  SCSI target IDs  0 1 2 3 4 5 6 7, all LUNs
Scanning host 5 for  SCSI target IDs  0 1 2 3 4 5 6 7, all LUNs
Scanning host 6 for  all SCSI target IDs, all LUNs
 Scanning for device 6 0 0 2 ...           
OLD: Host: scsi6 Channel: 00 Id: 00 Lun: 02
     Vendor: COMPELNT Model: Compellent Vol   Rev: 0701
     Type:   Direct-Access                    ANSI SCSI revision: 05
 Scanning for device 6 0 0 1 ... 
OLD: Host: scsi6 Channel: 00 Id: 00 Lun: 01
     Vendor: COMPELNT Model: Compellent Vol   Rev: 0602
     Type:   Direct-Access                    ANSI SCSI revision: 05
Scanning for device 6 0 1 2 ...           
OLD: Host: scsi6 Channel: 00 Id: 01 Lun: 02
     Vendor: COMPELNT Model: Compellent Vol   Rev: 0701
     Type:   Direct-Access                    ANSI SCSI revision: 05
Scanning for device 6 0 1 1 ... 
OLD: Host: scsi6 Channel: 00 Id: 01 Lun: 01
     Vendor: COMPELNT Model: Compellent Vol   Rev: 0602
     Type:   Direct-Access                    ANSI SCSI revision: 05
Scanning host 7 for  all SCSI target IDs, all LUNs
 Scanning for device 7 0 6 1 ...           
OLD: Host: scsi7 Channel: 00 Id: 06 Lun: 01
     Vendor: COMPELNT Model: Compellent Vol   Rev: 0602
     Type:   Direct-Access                    ANSI SCSI revision: 05
Scanning for device 7 0 6 2 ... 
OLD: Host: scsi7 Channel: 00 Id: 06 Lun: 02
     Vendor: COMPELNT Model: Compellent Vol   Rev: 0701
     Type:   Direct-Access                    ANSI SCSI revision: 05
Scanning for device 7 0 7 1 ...           
OLD: Host: scsi7 Channel: 00 Id: 07 Lun: 01
     Vendor: COMPELNT Model: Compellent Vol   Rev: 0602
     Type:   Direct-Access                    ANSI SCSI revision: 05
Scanning for device 7 0 7 2 ... 
OLD: Host: scsi7 Channel: 00 Id: 07 Lun: 02
     Vendor: COMPELNT Model: Compellent Vol   Rev: 0701
     Type:   Direct-Access                    ANSI SCSI revision: 05
Scanning host 9 for  SCSI target IDs  0 1 2 3 4 5 6 7, all LUNs
 Scanning for device 9 2 0 0 ...           
OLD: Host: scsi9 Channel: 02 Id: 00 Lun: 00
     Vendor: LSI      Model: MR9286-8e        Rev: 3.29
     Type:   Direct-Access                    ANSI SCSI revision: 05
Scanning host 10 for  SCSI target IDs  0 1 2 3 4 5 6 7, all LUNs
 Scanning for device 10 2 0 0 ... 
OLD: Host: scsi10 Channel: 02 Id: 00 Lun: 00
     Vendor: LSI      Model: MR9261-8i        Rev: 2.13
     Type:   Direct-Access                    ANSI SCSI revision: 05
Scanning for device 10 2 1 0 ... 
OLD: Host: scsi10 Channel: 02 Id: 01 Lun: 00
     Vendor: LSI      Model: MR9261-8i        Rev: 2.13
     Type:   Direct-Access                    ANSI SCSI revision: 05
0 new or changed device(s) found.          
0 remapped or resized device(s) found.
0 device(s) removed.

With the snapshot added you get name conflicts on LVM

root@marvin:/storage/home/users/ramon > multipath -ll
mpathd (36000d3100051f800000000000000016e) dm-6 COMPELNT,Compellent Vol
size=62T features='1 queue_if_no_path' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
  |- 6:0:0:1  sde 8:64  active ready running
  |- 7:0:6:1  sdg 8:96  active ready running
  |- 6:0:1:1  sdf 8:80  active ready running
  `- 7:0:7:1  sdh 8:112 active ready running
mpathf (36000d3100051f80000000000000002e2) dm-8 COMPELNT,Compellent Vol
size=62T features='1 queue_if_no_path' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
  |- 6:0:0:2  sdd 8:48  active ready running
  |- 7:0:7:2  sdk 8:160 active ready running
  |- 6:0:1:2  sdi 8:128 active ready running
  `- 7:0:6:2  sdj 8:144 active ready running

Background information

Jason Cormie of IT Services supplied the following details

The volume is raided across 348 disks (mix of SSD,SATA, SAS with 29 being hot spares), and asynchronously mirrored to our secondary datacentre in Maths.

Data scrubbing runs in the background regularly to detect any bad blocks, before they become an issue.

Snapshots for rollback are also mirrored to the secondary DC.

We can do synchronous mirroring, but it would have a performance impact as you’d have to wait for commits from the far side every time you wrote.

Note: synchronous mirroring is not enabled.