How to Prepare Linux Servers for Oracle RAC 10g

Information about How To Configure Linux Servers for Oracle RAC, 10G.

This document describes how to perform pre-installation tasks for Oracle RAC Servers. This includes, configuring /u00 filesystems, running delta scripts for SAN and Oracle, connecting the fiber, syncing up the SAN LUNs, configuring rawdevices, and finally setting up ssh for Oracle user account.

Document Creator Todd Walters

Overview of Pre-Installation Tasks to Install Oracle RAC 10g

1. Perform Post-Kickstart Configuration Task first
2. The /u00 filesystem is now created with Kickstart, but verify it's correct and in /etc/fstab
3. Skip the 'Create the /backup filesystem'. Will configure at a later date.
4. Reboot Server
5. After server is back up, plug in fiber to both HBA's
6. Setup Rawdevices for Oracle ( see below)
7. Go to How To Configure Oracle RAC OCR & Voting Disks
8. Configure ssh keys for Oracle User (Perform on Node1, if rebuilding other nodes, create key on new node then update on Node1)
9. Hand off to Oracle DBA to begin Installation
10. Return to Configure Oracle ASM Section when it's ready to install.
11. Hand back off to Oracle DBA to Finish.

Optional Step for Creating /u00 Filesystem, if not already there.

Steps for setting up /u00 and backup and creating Volume Group on Database servers

1. Use parted to show available space.

[root@srv0dbx03 Linux]# parted /dev/sda print

Disk geometry for /dev/sda: 0.000-208128.000 megabytes

Disk label type: msdos

Minor    Start       End     Type      Filesystem  Flags

1          0.031    101.975  primary   ext3        boot

2        101.975  14441.242  primary   ext3

3      14441.243  28780.510  primary   ext3

4      28780.510 208123.330  extended

5      28780.541  43119.777  logical   ext3

6      43119.809  57459.045  logical   ext3

7      57459.076  61553.737  logical   ext3

8      61553.769  63546.174  logical   linux-swap

Information: Don't forget to update /etc/fstab, if necessary.

2. Here we see there is space after 63546.174 so we will use this to add VolumeGroup[root@srv0dbx03 Linux]# parted /dev/sda "mkpart logical 63546.175 208123.330"
   Information: Don't forget to update /etc/fstab, if necessary.

3. Verify logical partition was created 

[root@srv0dbx03 Linux]# parted /dev/sda print

Disk geometry for /dev/sda: 0.000-208128.000 megabytes

Disk label type: msdos

Minor    Start       End     Type      Filesystem  Flags

1          0.031    101.975  primary   ext3        boot

2        101.975  14441.242  primary   ext3

3      14441.243  28780.510  primary   ext3

4      28780.510 208123.330  extended

5      28780.541  43119.777  logical   ext3

6      43119.809  57459.045  logical   ext3

7      57459.076  61553.737  logical   ext3

8      61553.769  63546.174  logical   linux-swap

9      63546.205 208123.330  logical

Information: Don't forget to update /etc/fstab, if necessary.

[root@srv0dbx03 dev]# lvm pvs

  PV         VG            Fmt  Attr PSize   PFree

  /dev/sda9  srv0dbx03_localVG lvm2 a-   141.19G 141.19G

  /dev/sdm   homeVG        lvm2 a-   133.21G 212.00M

4. Create the new physical volume on the new partition [root@srv0dbx03 Linux]# lvm pvcreate /dev/sda9
     Physical volume "/dev/sda9" successfully created

 Add this to the /etc/lvm/lvm.conf file

filter = [ "a/emcpower/",  "a/sdb$/", "a/sdb[0-9]/", "r|.*|" ]

5. Create a new volume group for a new logical volume[root@srv0dbx03 dev]# lvm vgcreate srv0dbx03_localVG /dev/sda9
   Volume group "srv0dbx03_localVG" successfully created

            * Note that new volume groups must be activated using the lvm vgchange -a y command:

            [root@srv0dbx03 dev]# lvm vgchange -a y srv0dbx03_localVG

• logical volume(s) in volume group "srv0dbx03_localVG" now active

6.  Create the logical volume for /u00

[root@srv0dbx03 dev]# lvm vgdisplay srv0dbx03_localVG

  --- Volume group ---

  VG Name               srv0dbx03_localVG

  System ID

  Format                lvm2

  Metadata Areas        1

  Metadata Sequence No  1

  VG Access             read/write

  VG Status             resizable

  MAX LV                0

  Cur LV                0

  Open LV               0

  Max PV                0

  Cur PV                1

  Act PV                1

  VG Size               141.19 GB

  PE Size               4.00 MB

  Total PE              36144

  Alloc PE / Size       0 / 0

  Free  PE / Size       36144 / 141.19 GB

  VG UUID               c8V4dj-6aMi-zcvc-OHJv-eTqE-wNBR-Byg35I

                                   

                        [root@srv0dbx03 dev]# lvm lvcreate -L 20g srv0dbx03_localVG -n u00

                          Logical volume "u00" created

                        [root@srv0dbx03 dev]# lvm lvs

  LV     VG            Attr   LSize   Origin Snap%  Move Log Copy%

  u00    srv0dbx03_localVG -wi-a-  20.00G

  homeLV homeVG        -wi--- 133.00G

[root@srv0dbx03 dev]# ls -l /dev/srv0dbx03_localVG/

total 0

lrwxrwxrwx  1 root root 29 Aug 25 13:43 u00 -> /dev/mapper/srv0dbx03_localVG-u00

7. Create a filesystem on the new logical volume:

[root@srv0dbx03 dev]# mkfs -t ext3 /dev/srv0dbx03_localVG/u00

mke2fs 1.35 (28-Feb-2004)

Filesystem label=

OS type: Linux

Block size=4096 (log=2)

Fragment size=4096 (log=2)

2621440 inodes, 5242880 blocks

262144 blocks (5.00%) reserved for the super user

First data block=0

Maximum filesystem blocks=4294967296

160 block groups

32768 blocks per group, 32768 fragments per group

16384 inodes per group

Superblock backups stored on blocks:

        32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,

        4096000

Writing inode tables: done

Creating journal (8192 blocks): done

Writing superblocks and filesystem accounting information: done

This filesystem will be automatically checked every 24 mounts or

180 days, whichever comes first.  Use tune2fs -c or -i to override.

8. Mount the new filesystem

[root@srv0dbx03 dev]# mount /dev/srv0dbx03_localVG/u00 /u00

[root@srv0dbx03 dev]# df -h /u00

Filesystem            Size  Used Avail Use% Mounted on

/dev/mapper/srv0dbx03_localVG-u00                       20G   76M   19G   1% /u00

9. Add the fileystem to the /etc/vstab file
   
   /dev/vg_srv0dbx01/lv_u00    /u00                    ext3    defaults        0 0

….end /u00 Filesystem Tasks (optional)

Begin Create Backup Logical Volume for /backup

1.  Create 4gb /backup logical volume

[root@srv0dbx03 dev]# lvcreate -L 4g srv0dbx03_localVG -n backup

  Logical volume "backup" created

  LV     VG            Attr   LSize   Origin Snap%  Move Log Copy%

  backup srv0dbx03_localVG -wi-a-   4.00G

  u00    srv0dbx03_localVG -wi-ao  20.00G

  homeLV homeVG        -wi--- 133.00G

2. Create /backup filesysem

[root@srv0dbx03 dev]# mkfs -t ext3 /dev/srv0dbx03_localVG/backup

mke2fs 1.35 (28-Feb-2004)

Filesystem label=

OS type: Linux

Block size=4096 (log=2)

Fragment size=4096 (log=2)

524288 inodes, 1048576 blocks

52428 blocks (5.00%) reserved for the super user

First data block=0

Maximum filesystem blocks=1073741824

32 block groups

32768 blocks per group, 32768 fragments per group

16384 inodes per group

Superblock backups stored on blocks:

        32768, 98304, 163840, 229376, 294912, 819200, 884736

Writing inode tables: done

Creating journal (8192 blocks): done

Writing superblocks and filesystem accounting information: done

This filesystem will be automatically checked every 27 mounts or

180 days, whichever comes first.  Use tune2fs -c or -i to override.

3. Mount /backup filesystem

[root@srv0dbx03 dev]# mkdir /backup

[root@srv0dbx03 dev]# mount /dev/srv0dbx03_localVG/backup /backup

[root@srv0dbx03 dev]# df -h /backup

Filesystem            Size  Used Avail Use% Mounted on

/dev/mapper/srv0dbx03_localVG-backup                      4.0G   41M  3.7G   2% /backup

4. Add the filesystems to /etc/fstab

/dev/srv0dbx03_localVG-backup /backup        ext3    defaults,acl    0 0 

Begin Delta Script for Oracle Servers, to Modify Kernel and install Oracle User

1. Run Oracle Delta script for Oracle servers.
• Run oracle10gR2_rhel4U5_delta
  # /home/admins/cci/sbin/oracle10gR2_rhel4U5_delta
• This script does the following:
• Creates oracle account and dba group.
• Modifies kernel tuning parameters (requires reboot) - Review script to see kernel modifications.
• Creates a set of empty files owned by oracle:dba (/etc/listner.ora, for example)
2. Change Ownership of filesystems to oracle:dba after running  oracle delta script.

[root@srv0dbx03 /]# chown -R oracle:dba /u00

[root@srv0dbx03 /]# chown -R oracle:dba /backup

3. Copy Oracle User Environment to /export/home/oracle

[root@srv0dbx02 /]# cd /home/admins/ora-inst/oracle
[root@srv0dbx02 /]# cp -vaR . /export/home/oracle

4. Reboot Server for Kernel changes to take effect.
5. After Reboot, Perform Prerequisites before beginning SAN script install

Requirement

1. Plug fiber into HBA's before starting this section
2. Re-connect appropriate storage group to approiate host i.e (srv0dbx02-sg to host srv0dbx02 in Navisphere Console)
3. To perform #2, login to 10.67.2.91 as admin/cox4usa and Connect Hosts to storage group.
• Only one host per storage group.

…end requirement

Begin SAN Install Script and LUN Configuration

1. Run SAN Delta script for servers connected to SAN
• [root@srv0dbx03:] $ cd /home/admins/cci/sbin
• [root@srv0dbx03:] $ ./san_delta_rhel4U5
• be sure to modify license section in san delta with B6P9-BB4M-BFQQ-QFOQ-MF9A-JICY
• or run; [root@srv0dbx02 sbin]# emcpreg -add B6P9-BB4M-BFQQ-QFOQ-MF9A-JICY
                           1 key(s) successfully added
2.  NOTE – Make sure to automate naviagent
•  [root@srv0dbx03 init.d]# chkconfig naviagent on
• [root@srv0dbx03 init.d]# chkconfig --list naviagent
• naviagent       0:off   1:off   2:off   3:on    4:on    5:on    6:off
3. Start Navisphere Agent on Server
             [root@srv0dbx02 init.d]# /etc/init.d/naviagent start
4. Perform lun scan to find all luns.   
               [root@srv0dbx02 init.d]# /opt/cci/sbin/lun_scan.sh all
5. Run PowerPath to Verify LUNS are there
             [root@srv0dbx02 sbin]# /etc/init.d/PowerPath start
             [root@srv0dbx02 init.d]# powermt display dev=all | more

             

6. Run emcpsync from /home/admins/cci/sbin/emcpsync to Sync up LUNs.
• use emcpinv_srv01srv0dbx01 LUN Inventory file
• [root@srv0dbx02 sbin]# ./emcpsync /home/admins/ora-inst/emcpinv_srv0dbx01.corp.cox.com
• Run until output matches below:
  [root@srv0dbx02 sbin]# ./emcpsync emcpinv_srv0dbx01.corp.cox.com
  Nothing to do for emcpowerh
  Nothing to do for emcpowerd
  Nothing to do for emcpoweri
  Nothing to do for emcpowerf
  Nothing to do for emcpowerc
  Nothing to do for emcpowerb
  Nothing to do for emcpowera
  Nothing to do for emcpowere
  Nothing to do for emcpowerg
7. Make sure LUNS are assigned as below:

[root@srv0dbx02 sbin]# cat emcpinv_srv0dbx01.corp.cox.com
#######################################################################
# emcpinv
# Timestamp: Fri Dec 19 10:43:13 2008
# Hostname: srv0dbx01.corp.cox.com
# Pseudo|Logical Dev ID
#######################################################################
emcpowera|60060160F5701D0022F0D4676C74DD11 [LUN 0]
emcpowerb|60060160F5701D0023F0D4676C74DD11 [LUN 1]
emcpowerc|60060160F5701D0024F0D4676C74DD11 [LUN 2]
emcpowerd|60060160F5701D0025F0D4676C74DD11 [LUN 3]
emcpowere|60060160F5701D0026F0D4676C74DD11 [LUN 4]
emcpowerf|60060160F5701D00A89D50024E80DD11 [LUN 30]
emcpowerg|60060160F5701D00DA42E9BC4379DD11 [LUN 10]
emcpowerh|60060160F5701D0052DD912E4479DD11 [LUN 21]
emcpoweri|60060160F5701D009ECAE24C4E80DD11 [LUN 41]
emcpowerj|60060160F5701D00A808FBB64479DD11 [LUN 50]
emcpowerk|60060160F5701D00CA01D4603EA0DD11 [LUN 101]

Begin Configure of RAW Devices

1. cd /etc/sysconfig/and edit with vi, the 'rawdevices'  file to match below.
   # Voting Disks
   /dev/raw/raw1   /dev/emcpowera
   /dev/raw/raw2   /dev/emcpowerb
   /dev/raw/raw3   /dev/emcpowerc
   # OCR disks
   /dev/raw/raw4   /dev/emcpowerd
   /dev/raw/raw5   /dev/emcpowere
2. Start rawdevices:
   
   [root@srv0dbx03 sysconfig]# service rawdevices start
   Assigning devices:
   Assigning devices:
              /dev/raw/raw1  -->   /dev/emcpowera
   /dev/raw/raw1:  bound to major 120, minor 0
              /dev/raw/raw2  -->   /dev/emcpowerb
   /dev/raw/raw2:  bound to major 120, minor 16
              /dev/raw/raw3  -->   /dev/emcpowerc
   /dev/raw/raw3:  bound to major 120, minor 32
              /dev/raw/raw4  -->   /dev/emcpowerd
   /dev/raw/raw4:  bound to major 120, minor 48
              /dev/raw/raw5  -->   /dev/emcpowere
   /dev/raw/raw5:  bound to major 120, minor 64
   done
3. Check Status of raw devices

[root@srv0dbx03 sysconfig]# service rawdevices status

/dev/raw/raw1:  bound to major 120, minor 0

/dev/raw/raw2:  bound to major 120, minor 16

/dev/raw/raw3:  bound to major 120, minor 32

/dev/raw/raw4:  bound to major 120, minor 48

/dev/raw/raw5:  bound to major 120, minor 64

4. Verify rawdevices is set to start at boot time.
   
   [root@srv0dbx02 sysconfig]# chkconfig --list rawdevices
   rawdevices      0:off   1:off   2:on    3:on    4:on    5:on    6:off

Begin SSH Key Configuration and .rhosts for Oracle User account

SSH Keys Installation Overview

• The ssh keys are generated on each host.
• Then on Node 1 you copy the keys from every host to the Authorized Keys file on node 1.
• Then copy the keys file to a central location to allow easier copying to the other nodes.
• Finally you copy the Authorized_keys file to each node and place in /export/home/oracle/.ssh
• Also, /export/home/oracle permission cannot be higher than 755.

1. Login as the Oracle user
   
    [root@srv0dbx01 sbin]# su - oracle
2. Create the /export/home/oracle/.ssh directory if it does not exist.

mkdir ~/.ssh
chmod 700 ~/.ssh

3. Generate a rsa and dsa key ( Do this on each node in the cluster)
   
   > /usr/bin/ssh-keygen -t rsa
   >/usr/bin/ssh-keygen -t dsa
4. Change directory to /export/home/oracle/.ssh
   
   cd /export/home/oracle.ssh
5. Add Keys to authorized_keys file on Node 1 (srv0dbx01)
   
   Example: copy contents of ~/.ssh/id_rsa.pub and ~/.ssh/id_dsa.pub to authorized_keys
   ssh srv0dbx01 cat /export/home/oracle/.ssh/id_rsa.pub >> authorized_keys
   ssh srv0dbx01 cat /export/home/oracle/.ssh/id_dsa.pub >> authorized_keys
   ssh srv0dbx02 cat /export/home/oracle/.ssh/id_rsa.pub >> authorized_keys
   ssh srv0dbx02 cat /export/home/oracle/.ssh/id_dsa.pub >> authorized_keys 
6. Copy authorized_keys file to central location.
   
   cp ~/.ssh/authorized_keys /home/admins/ora-inst/oracle/.ssh/
7. Copy central authorized_keys file from Node 1 to all nodes in the cluster.
   
   cp /home/admins/ora-inst/oracle/.ssh/authorized_keys  /export/home/oracle/.ssh/.
8. Change Permissions on Oracle users authorized_keys file. (If Necessary)
   
   Persmissions should be 600 or rw------- for authorized_keys file. If not, do this:
   chmod 600 authorized_keys
9. Test ssh keys from node you will be doing the installation on (You should not be prompted for a password)

Example: ssh srvsrv0dbx02 hostname (Repeat for all nodes in cluster)

10. Create .rhosts file
    
    touch /export/home/oracle/.rhosts and add node information.
    > more .rhosts
    srv0dbx01 oracle
    srv0dbx02 oracle
    srv0dbx03 oracle
    srv0dbx01-vip oracle
    srv0dbx02-vip oracle
    srv0dbx03-vip oracle
    srv0dbx01-ic oracle
    srv0dbx02-ic oracle
    srv0dbx03-ic oracle

Complete Pre-Insallation Tasks for Oracle Servers 

1. At this point, hand off to DBA and instruct them to begin installation on page 15 of this file:
   Linux-Oracle10g Installation Standards
2. When DBA completes installation you will then be required to configure ASM Storage for RAC. .
   Use How To Configure ASM for Oracle Servers document to configure ASM 
   Then the DBA will continue with install Databases