Information about How to Reconfigure ASM Disks on Virtual Servers
This document will describe how to relocate OCR and Voting Disks on RAC Virtual Servers. We will relocate the OCR & Voting disks from one Datastore to another.
Document Creator Todd Walters
Reason for OCR and Voting disk Relocation
Was originally built using 15k 73gb FC Drives on EMC SAN
Dev & Test required more storage for dwak/twak
There were empty SATA Slots on existing DAE
Ordered 6 1tb 7200 SATA Drives and create new Raid Group 102 R5
Overview of OCR & Voting Disk Relocation
Relocate during scheduled downtime to remove the old 55gb ASM vmdk disks from each virtual server
During downtime relocate OCR and Voting Storage vmdk files to new SATA Datastore
Update SAN Storage Layout diagram and create/update any relevant how-to documentation.
New datastore path = /vmfs/volumes/cx3c-1lun102/shared with dev and test sub-directories below.
-
How to Relocate Existing OCR & Voting Disks
Shutdown Oracle RAC Dev or Test using crs_stop -all
Remove the four 55gb ASM disks from all 3 nodes using vSphere client.
Prior to moving the vmdk files, write down the scsi id:x layout for all asm,ocr, & voting disks.
disk 1 = SCSI (0:0) will always be OS Disk
example layout (OCR, voting, and ASM Backup)disk 2 = SCSI (1:0) - Voting 1
disk 3 = SCSI (1:1) - Voting 2
disk 8 = SCSI (1:2) - Voting 3
disk 9 = SCSI (1:3) - OCR 1
disk 10 = SCSI (1:4) - OCR 2
disk 11 = SCSI (1:5) - ASM Backup
standard layout (node 3)disk 2 = SCSI (1:0) - Voting 1
disk 3 = SCSI (1:1) - Voting 2
disk 8 = SCSI (1:2) - Voting 3
disk 9 = SCSI (1:3) - OCR 1
disk 10 = SCSI (1:4) - OCR 2
disk 4 = SCSI (1:13) - XASM1 Disk
disk 5 = SCSI (1:14) - XASM2 Disk
disk 11 = SCSI (1:5) - ASM Backup Disk
disk 6 = SCSI (1:15) - NFS Backup Disk
Manually move (since no storage vmotion on esxi) the 3 Voting and 2 OCR disks from old 73gb 15k disk datastore to the new 1tb disk drive datastore.
Login to ESXi host via ssh/putty
change directory to /vmfs/volumes/emc-cx3-1lun100/shared (for test)
run following command to list flat files and size (remember there are other vmdk files w/out -flat)
run the following command to move all the vmdk's to new location from old cx3-1lun100
mv *.vmdk /vmfs/volumes/emc-cx3-1lun102/shared/test/.
note: In this step we will relocate the existing OCR & Voting disks to a different datastore. This will require the vm to be powered off to add the virtual hard disk. Once ESXi hosts are licensed this will not be required.
Verify all files moved properly by changing directory to new location and running following command
Move the ASM backup 50gb vmdk to the 1tb storage area as well.
Use Datastore Browser to move vmdk or cmd line command.
Change directory to /vmfs/volumes/lesxit01-vmfs02/oracle/shared
run - mv *.vmdk /vmfs/volumes/emc-cx3-1lun102/shared/test/.
Manually remove all old ASM virtual disks from each node.
Add each voting, ocr, and the single nfs backup disk to each node. Carefully making sure the right virtual scsi controller and ID. use SCSI (1:5) for ASM_Backup (matching standard layout above)
Bring one node online at a time and verify everything is online. Check Oracle status using 'showcrs'
Verify with DBA that everything was online for RAC
Once everything is verify, remove the emc-cx3-1lun100 datastore (73gb 15k drives)from ESXi Test R710 via vSphere Client
Login to NaviSphere and remove disk group
Follow How to Remove Storage from SAN guide
Unbind LUN, Remove from Storage Group, Destroy Raid Group.
Disk Layout Paths
There are some minor cleanup with the paths. All devices past the OCR/Voting disk path changed. For example, the first ASM Disk changed from /dev/sdl1 to /dev/sdh1
Here's output of all the old disks:
[root@csrv0dbxd11 ~]# fdisk -l | grep Disk
Disk /dev/sda: 53.6 GB, 53687091200 bytes
Disk /dev/sdb: 268 MB, 268435456 bytes
Disk /dev/sdc: 268 MB, 268435456 bytes
Disk /dev/sdd: 268 MB, 268435456 bytes
Disk /dev/sde: 268 MB, 268435456 bytes
Disk /dev/sdf: 268 MB, 268435456 bytes
Disk /dev/sdg: 42.9 GB, 42949672960 bytes
Disk /dev/sdh: 59.0 GB, 59055800320 bytes
Disk /dev/sdi: 59.0 GB, 59055800320 bytes
Disk /dev/sdj: 59.0 GB, 59055800320 bytes
Disk /dev/sdk: 59.0 GB, 59055800320 bytes
Disk /dev/sdl: 214.7 GB, 214748364800 bytes
Disk /dev/sdm: 214.7 GB, 214748364800 bytes
New disks (well disk paths)
[root@csrv0dbxd11 init.d]# fdisk -l | grep Disk
Disk /dev/sda: 53.6 GB, 53687091200 bytes
Disk /dev/sdb: 268 MB, 268435456 bytes
Disk /dev/sdc: 268 MB, 268435456 bytes
Disk /dev/sdd: 268 MB, 268435456 bytes
Disk /dev/sde: 268 MB, 268435456 bytes
Disk /dev/sdf: 268 MB, 268435456 bytes
Disk /dev/sdg: 42.9 GB, 42949672960 bytes
Disk /dev/sdh: 214.7 GB, 214748364800 bytes
Disk /dev/sdi: 214.7 GB, 214748364800 bytes
Voting & OCR paths stayed the same
# Voting Disks
/dev/raw/raw1 /dev/sdb1
/dev/raw/raw2 /dev/sdc1
/dev/raw/raw3 /dev/sdd1
# OCR disks
/dev/raw/raw4 /dev/sde1
/dev/raw/raw5 /dev/sdf1
Old Datastore path
/vmfs/volumes/emc-cx3-1lun101/shared # ls -l | awk ' { print $9 }'
asm01-flat.vmdk
asm01.vmdk
asm02-flat.vmdk
asm02.vmdk
asm03-flat.vmdk
asm03.vmdk
asm04-flat.vmdk
asm04.vmdk
ocr1-flat.vmdk
ocr1.vmdk
ocr2-flat.vmdk
ocr2.vmdk
voting1-flat.vmdk
voting1.vmdk
voting2-flat.vmdk
voting2.vmdk
voting3-flat.vmdk
voting3.vmdk
new Datastore path
/vmfs/volumes/cx3c-1lun102/shared/dev # ls -l | awk ' { print $9 }'
backup_asm01-flat.vmdk
backup_asm01.vmdk
dasm1-flat.vmdk
dasm1.vmdk
dasm2-flat.vmdk
dasm2.vmdk
ocr1-flat.vmdk
ocr1.vmdk
ocr2-flat.vmdk
ocr2.vmdk
voting1-flat.vmdk
voting1.vmdk
voting2-flat.vmdk
voting2.vmdk
voting3-flat.vmdk
voting3.vmdk
