How to Relocate OCR and Voting Disks on Virtual Servers

Information about How to Reconfigure ASM Disks on Virtual Servers

This document will describe how to relocate OCR and Voting Disks on RAC Virtual Servers. We will relocate the OCR & Voting disks from one Datastore to another.

Document Creator   Todd Walters

Reason for OCR and Voting disk Relocation

1. Was originally built using 15k 73gb FC Drives on EMC SAN
2. Dev & Test required more storage for dwak/twak
3. There were empty SATA Slots on existing DAE
4. Ordered 6 1tb 7200 SATA Drives and create new Raid Group 102 R5

Overview of OCR & Voting Disk Relocation

• Relocate during scheduled downtime to remove the old 55gb ASM vmdk disks from each virtual server
• During downtime relocate OCR and Voting Storage vmdk files to new SATA Datastore
• Update SAN Storage Layout diagram and create/update any relevant how-to documentation.
• New datastore path = /vmfs/volumes/cx3c-1lun102/shared with dev and test sub-directories below.
• See How to Reconfigure ASM Disks on Virtual Servers  for additional information.

How to Relocate Existing OCR & Voting Disks

1. Shutdown Oracle RAC Dev or Test using crs_stop -all
2. Remove the four 55gb ASM disks from all 3 nodes using vSphere client.
3. Prior to moving the vmdk files, write down the scsi id:x layout for all asm,ocr, & voting disks.
• disk 1 = SCSI (0:0) will always be OS Disk
• example layout (OCR, voting, and ASM Backup)disk 2 = SCSI (1:0) - Voting 1
  disk 3 = SCSI (1:1) - Voting 2
  disk 8 = SCSI (1:2) - Voting 3
  disk 9 = SCSI (1:3) - OCR 1
  disk 10 = SCSI (1:4) - OCR 2
  disk 11 = SCSI (1:5) - ASM Backup
• standard layout (node 3)disk 2 = SCSI (1:0) - Voting 1
  disk 3 = SCSI (1:1) - Voting 2
  disk 8 = SCSI (1:2) - Voting 3
  disk 9 = SCSI (1:3) - OCR 1
  disk 10 = SCSI (1:4) - OCR 2
  disk 4 = SCSI (1:13) - XASM1 Disk
  disk 5 = SCSI (1:14) - XASM2 Disk
  disk 11 = SCSI (1:5) - ASM Backup Disk
  disk 6 = SCSI (1:15) - NFS Backup Disk
4. Manually move (since no storage vmotion on esxi) the 3 Voting and 2 OCR disks from old 73gb 15k disk datastore to the new 1tb disk drive datastore.

• Login to ESXi host via ssh/putty
• change directory to /vmfs/volumes/emc-cx3-1lun100/shared (for test)
• run following command to list flat files and  size (remember there are other vmdk files w/out -flat)
• #  ls -lh | grep flat | awk '{ print $5,$9 }' 55.0G asm1-flat.vmdk
  55.0G asm2-flat.vmdk
  55.0G asm3-flat.vmdk
  55.0G asm4-flat.vmdk
  250.0M ocr1-flat.vmdk
  250.0M ocr2-flat.vmdk
  250.0M voting1-flat.vmdk
  250.0M voting2-flat.vmdk
  250.0M voting3-flat.vmdk
• run the following command to move all the vmdk's to new location from old cx3-1lun100
• mv *.vmdk /vmfs/volumes/emc-cx3-1lun102/shared/test/.
• note: In this step we will relocate the  existing OCR & Voting disks to a different datastore. This will require the vm to be powered off to add the virtual hard disk.  Once ESXi hosts are licensed this will not be required.
• Verify all files moved properly by changing directory to new location and running following command
•  ls -lh | awk '{ print $5,$9 }'
• Move the ASM backup 50gb vmdk to the 1tb storage area as well.
• Use Datastore Browser to move vmdk or cmd line command.
• Change directory to /vmfs/volumes/lesxit01-vmfs02/oracle/shared
• run - mv *.vmdk /vmfs/volumes/emc-cx3-1lun102/shared/test/.
• Manually remove all old ASM virtual disks from each node.
• Login to vSphere Client and power off all 3 nodes
• Edit the settings of each node and remove all 55gb ASM Disks
• Add each voting, ocr, and the single nfs backup disk to each node. Carefully making sure the right virtual scsi controller and ID. use SCSI (1:5) for ASM_Backup (matching standard layout above)
• Bring one node online at a time and verify everything is online. Check Oracle status using 'showcrs'
• Verify with DBA that everything was online for RAC
• Once everything is verify, remove the emc-cx3-1lun100 datastore (73gb 15k drives)from ESXi Test R710 via vSphere Client
• Login to NaviSphere and remove disk group

• Follow How to Remove Storage from SAN guide
• Unbind LUN, Remove from Storage Group, Destroy Raid Group.

Disk Layout Paths

There are some minor cleanup with the paths. All devices past the OCR/Voting disk path changed. For example, the first ASM Disk changed from /dev/sdl1 to /dev/sdh1

1. Here's output of all the old disks:

[root@csrv0dbxd11 ~]# fdisk -l | grep Disk
Disk /dev/sda: 53.6 GB, 53687091200 bytes
Disk /dev/sdb: 268 MB, 268435456 bytes
Disk /dev/sdc: 268 MB, 268435456 bytes
Disk /dev/sdd: 268 MB, 268435456 bytes
Disk /dev/sde: 268 MB, 268435456 bytes
Disk /dev/sdf: 268 MB, 268435456 bytes
Disk /dev/sdg: 42.9 GB, 42949672960 bytes
Disk /dev/sdh: 59.0 GB, 59055800320 bytes
Disk /dev/sdi: 59.0 GB, 59055800320 bytes
Disk /dev/sdj: 59.0 GB, 59055800320 bytes
Disk /dev/sdk: 59.0 GB, 59055800320 bytes
Disk /dev/sdl: 214.7 GB, 214748364800 bytes
Disk /dev/sdm: 214.7 GB, 214748364800 bytes

New disks (well disk paths)

[root@csrv0dbxd11 init.d]# fdisk -l | grep Disk
Disk /dev/sda: 53.6 GB, 53687091200 bytes
Disk /dev/sdb: 268 MB, 268435456 bytes
Disk /dev/sdc: 268 MB, 268435456 bytes
Disk /dev/sdd: 268 MB, 268435456 bytes
Disk /dev/sde: 268 MB, 268435456 bytes
Disk /dev/sdf: 268 MB, 268435456 bytes
Disk /dev/sdg: 42.9 GB, 42949672960 bytes
Disk /dev/sdh: 214.7 GB, 214748364800 bytes
Disk /dev/sdi: 214.7 GB, 214748364800 bytes

Voting & OCR paths stayed the same

# Voting Disks
/dev/raw/raw1   /dev/sdb1
/dev/raw/raw2   /dev/sdc1
/dev/raw/raw3   /dev/sdd1
# OCR disks
/dev/raw/raw4   /dev/sde1
/dev/raw/raw5   /dev/sdf1

Old Datastore path

/vmfs/volumes/emc-cx3-1lun101/shared # ls -l | awk ' { print $9 }'
asm01-flat.vmdk
asm01.vmdk
asm02-flat.vmdk
asm02.vmdk
asm03-flat.vmdk
asm03.vmdk
asm04-flat.vmdk
asm04.vmdk
ocr1-flat.vmdk
ocr1.vmdk
ocr2-flat.vmdk
ocr2.vmdk
voting1-flat.vmdk
voting1.vmdk
voting2-flat.vmdk
voting2.vmdk
voting3-flat.vmdk
voting3.vmdk

new Datastore path

/vmfs/volumes/cx3c-1lun102/shared/dev # ls -l | awk ' { print $9 }'
backup_asm01-flat.vmdk
backup_asm01.vmdk
dasm1-flat.vmdk
dasm1.vmdk
dasm2-flat.vmdk
dasm2.vmdk
ocr1-flat.vmdk
ocr1.vmdk
ocr2-flat.vmdk
ocr2.vmdk
voting1-flat.vmdk
voting1.vmdk
voting2-flat.vmdk
voting2.vmdk
voting3-flat.vmdk
voting3.vmdk