#########################
SIMPLE WAY TO FAKE DISKS
#########################
Create Files using "dd"
# mkdir /asmdisks
# chown oracle:dba /asmdisks
# su - oracle
$ dd if=/dev/zero of=/asmdisks/_file_disk1 bs=1k count=100000
$ dd if=/dev/zero of=/asmdisks/_file_disk2 bs=1k count=100000
$ dd if=/dev/zero of=/asmdisks/_file_disk3 bs=1k count=100000
$ dd if=/dev/zero of=/asmdisks/_file_disk4 bs=1k count=100000
We now have four files to be used as virtual disks of 100MB each:
$ ls -l /asmdisks
total
400432
-rw-r--r-- 1 oracle dba 102400000 Jun 8 15:30 _file_disk1
-rw-r--r-- 1 oracle dba 102400000 Jun 8 15:31 _file_disk2
-rw-r--r-- 1 oracle dba 102400000 Jun 8 15:31 _file_disk3
-rw-r--r-- 1 oracle dba 102400000 Jun 8 15:31 _file_disk4
Associate Loop Device with the New Files
This needs to be performed as the root user account:
# losetup /dev/loop1 /asmdisks/_file_disk1
# losetup /dev/loop2 /asmdisks/_file_disk2
# losetup /dev/loop3 /asmdisks/_file_disk3
# losetup /dev/loop4 /asmdisks/_file_disk4
Bind RAW Devices to Existing Block Devices
As the root user account:
# raw /dev/raw/raw1 /dev/loop1/dev/raw/raw1: bound to major 7, minor 1
# raw /dev/raw/raw2 /dev/loop2/dev/raw/raw2: bound to major 7, minor 2
# raw /dev/raw/raw3 /dev/loop3/dev/raw/raw3: bound to major 7, minor 3
# raw /dev/raw/raw4 /dev/loop4/dev/raw/raw4: bound to major 7, minor 4
Change Ownership of RAW Devices
change the ownership of all four RAW devices:
# chown oracle:dba /dev/raw/raw1
# chown oracle:dba /dev/raw/raw2
# chown oracle:dba /dev/raw/raw3
# chown oracle:dba /dev/raw/raw4
# chmod 660 /dev/raw/raw1
# chmod 660 /dev/raw/raw2
# chmod 660 /dev/raw/raw3
# chmod 660 /dev/raw/raw4
This completes creating the ASM disks on Linux. When we create database using dbca with ASM option, these disks would be visible.
Thursday, August 30, 2007
Monday, March 12, 2007
Install 10g R2 RAC on Solaris 10 06/06 using iSCSI and Vmware server 1.0.1
Install 10gR2 RAC on Solaris 10(06/06) using iSCSI & Vmware server 1.0.1
1. Install VMWARE SERVER 1.0 on windows 2003 advanced server.
2. Download openfiler iso image downloaded from www.openfiler.com. It is pretty much similar to the Redhat Linux installation.
3. Create a node using vmware and select operating system as Redhat Linux. Create a disk of 2gb size (To be used for creating iscsi volumes to store the CRS files and increase memory to 512mb. Edit the cdrom and select the ‘use iso image’ option and browse the openfiler iso downloaded as above. Click “start virtual machine” and the virtual machine will boot from the iso image as selected above.
Under volumes – physical storage management, click on the /dev/sdb (which is used to create the volumes).
Similarly, create the volumes for OCR, VOTING DISK, ASM files.
Once this is done, click on ‘edit’ under each volume i.e ocr,vot,asm etc and select ‘allow’ for both the ip addresses.
This completes the configuration part on the open filer. The iscsi volumes are created under volume group ‘MYRAC’ and the RAC nodes were given permission to access the volumes on the openfiler.
CONFIGURATION ON THE SOLARIS BOXES
1. Install solaris x86 update 3 (11/06) using vmware. Create new node with solaris 10 operating system option, 512 MB RAM, 7gb scsi hard disk for solaris and oracle 10g s/w installation (crs_home and oracle_home), add two nic cards(One for public ip and the other for private interconnect).
2. Enter the following values for the ip addresses for two nodes.
Solaris node one:
Pcn1 (first network card for public ip)
Hostname: sola1.micro.com
Ip address: 150.0.0.15
Netmask : 255.255.255.0
Pcn2 (second network card for private ip)
Hostname: sola1-prv.micro.com
Ip address: 14.0.0.14
Netmask : 255.255.255.0
Solaris second node:
Pcn1 (first network card)
Hostname: sola2.micro.com
Ip address: 150.0.0.16
Netmask : 255.255.255.0
Pcn2 (second network card)
Hostname: sola2-prv.micro.com
Ip address: 14.0.0.15
Netmask : 255.255.255.0
2. Once the solaris 10 x86 installation is completed on both the nodes, check the ip addresses (ifconfig –a) on both the boxes.
1. Install VMWARE SERVER 1.0 on windows 2003 advanced server.
2. Download openfiler iso image downloaded from www.openfiler.com. It is pretty much similar to the Redhat Linux installation.
3. Create a node using vmware and select operating system as Redhat Linux. Create a disk of 2gb size (To be used for creating iscsi volumes to store the CRS files and increase memory to 512mb. Edit the cdrom and select the ‘use iso image’ option and browse the openfiler iso downloaded as above. Click “start virtual machine” and the virtual machine will boot from the iso image as selected above.
While entering the values for the eth0, enter the values as per the private interconnect used for the rac nodes. I gave 14.0.0.5 for openfiler and the private IP’s for the RAC nodes are 14.0.0.14 and 14.0.0.15. The reason being the RAC nodes interact with the openfiler (iSCSI target) using private interconnect.
The following screenshots explain the procedure to create the SCSI volumes (LUNs) to be shared by the two RAC nodes.
Go to services and click enable for the iscsi target and status should become enabled.
Go to general – local networks and enter the values of the two solaris nodes with their private ip addresses.
Under volumes – physical storage management, click on the /dev/sdb (which is used to create the volumes).
Click volumes -- create new volume, to create new volume for the /dev/sdb. The mode and partition type should be as ‘primary’ and ‘physical volume’. Click ‘create’
Click volumes – volume group management and enter the value for volume group name (ex.. myrac) and select the physical volume /dev/sdb1. click ‘add volume group’
Click volumes – create new volume to create the volumes in the physical volume /dev/sdb1. select file system type as ‘iSCSI’.
Similarly, create the volumes for OCR, VOTING DISK, ASM files.
Once this is done, click on ‘edit’ under each volume i.e ocr,vot,asm etc and select ‘allow’ for both the ip addresses.
This completes the configuration part on the open filer. The iscsi volumes are created under volume group ‘MYRAC’ and the RAC nodes were given permission to access the volumes on the openfiler.
CONFIGURATION ON THE SOLARIS BOXES
1. Install solaris x86 update 3 (11/06) using vmware. Create new node with solaris 10 operating system option, 512 MB RAM, 7gb scsi hard disk for solaris and oracle 10g s/w installation (crs_home and oracle_home), add two nic cards(One for public ip and the other for private interconnect).
2. Enter the following values for the ip addresses for two nodes.
Solaris node one:
Pcn1 (first network card for public ip)
Hostname: sola1.micro.com
Ip address: 150.0.0.15
Netmask : 255.255.255.0
Pcn2 (second network card for private ip)
Hostname: sola1-prv.micro.com
Ip address: 14.0.0.14
Netmask : 255.255.255.0
Solaris second node:
Pcn1 (first network card)
Hostname: sola2.micro.com
Ip address: 150.0.0.16
Netmask : 255.255.255.0
Pcn2 (second network card)
Hostname: sola2-prv.micro.com
Ip address: 14.0.0.15
Netmask : 255.255.255.0
2. Once the solaris 10 x86 installation is completed on both the nodes, check the ip addresses (ifconfig –a) on both the boxes.
verify that the iSCSI software packages are installed.
# pkginfo SUNWisciu SUNWiscsir
system SUNWiscsiu Sun iSCSI Device Driver (root)
system SUNWiscsir Sun iSCSI Management Utilities (usr)
3. in /etc/hosts of both solaris boxes, enter the value of openfiler ip address. i.e 14.0.0.5.
/etc/hosts file on both the solaris boxes should like,
127.0.0.1 localhost.localdomain localhost
150.0.0.15 sola1.micro.com sola1
150.0.0.16 sola2.micro.com sola2
14.0.0.14 sola1-prv.micro.com sola1-prv
14.0.0.15 sola2-prv.micro.com sola2-prv
150.0.0.17 sola1-vip.micro.com sola1-vip
150.0.0.18 sola2-vip.micro.com sola2-vip
14.0.0.5 filer.micro.com filer
a) Determine if the iSCSI target device is reachable by using the telnet command to connect to the iSCSI target device using port 3260.
b) Configure the iSCSI device discovery method.
#iscsiadm add discovery-address 14.0.0.5:3260
if you want to remove the discover-address, use "disable" option. After this use "remove" to remove the discovery address.
#iscsiadm modify discovery-address 14.0.0.5:3260 disable
#iscsiadm modify discovery-address 14.0.0.5:3260 remove
The iSCSI connection is not initiated until the discovery method is enabled.
d) Enable the iSCSI target discovery method using the following:
# iscsiadm modify discovery --sendtargets enable
#devfsadm (Restart the devfsadm service)
e) Create the iSCSI device links for the local system.
# devfsadm -i iscsi
Once the above command is run, the iscsi volumes are discovered by both the solaris boxes and the same can be seen as follows.
#iscsiadm list target –S
# pkginfo SUNWisciu SUNWiscsir
system SUNWiscsiu Sun iSCSI Device Driver (root)
system SUNWiscsir Sun iSCSI Management Utilities (usr)
3. in /etc/hosts of both solaris boxes, enter the value of openfiler ip address. i.e 14.0.0.5.
/etc/hosts file on both the solaris boxes should like,
127.0.0.1 localhost.localdomain localhost
150.0.0.15 sola1.micro.com sola1
150.0.0.16 sola2.micro.com sola2
14.0.0.14 sola1-prv.micro.com sola1-prv
14.0.0.15 sola2-prv.micro.com sola2-prv
150.0.0.17 sola1-vip.micro.com sola1-vip
150.0.0.18 sola2-vip.micro.com sola2-vip
14.0.0.5 filer.micro.com filer
a) Determine if the iSCSI target device is reachable by using the telnet command to connect to the iSCSI target device using port 3260.
b) Configure the iSCSI device discovery method.
#iscsiadm add discovery-address 14.0.0.5:3260
if you want to remove the discover-address, use "disable" option. After this use "remove" to remove the discovery address.
#iscsiadm modify discovery-address 14.0.0.5:3260 disable
#iscsiadm modify discovery-address 14.0.0.5:3260 remove
The iSCSI connection is not initiated until the discovery method is enabled.
d) Enable the iSCSI target discovery method using the following:
# iscsiadm modify discovery --sendtargets enable
#devfsadm (Restart the devfsadm service)
e) Create the iSCSI device links for the local system.
# devfsadm -i iscsi
Once the above command is run, the iscsi volumes are discovered by both the solaris boxes and the same can be seen as follows.
#iscsiadm list target –S
The OS device names should be same on both the solaris boxes and connections value should be 1. This will confirm that iscsi volumes have been configured properly. The same can be verified from the web interface of the OpenFiler under status tab.
If the connections value is 0 or if any other error is encountered, check dmesg or /var/adm/messages.
The default port for iscsi is 3260. Try pinging to the iscsi target or telnet to the target on the port 3260.
After resolving the above errors, run the following
#devfsadm
#iscsiadm list target –S
This should give the output as above in the screenshot.
Also verify if the iscsi targets are available on both the solaris boxes using format command.
If the connections value is 0 or if any other error is encountered, check dmesg or /var/adm/messages.
The default port for iscsi is 3260. Try pinging to the iscsi target or telnet to the target on the port 3260.
After resolving the above errors, run the following
#devfsadm
#iscsiadm list target –S
This should give the output as above in the screenshot.
Also verify if the iscsi targets are available on both the solaris boxes using format command.
# format
Searching for disks...done
AVAILABLE DISK SELECTIONS:
0. c0d0
1. c2t1d0
2. c2t3d0
3. c2t3d0
4. c2t4d0
5. c2t5d0
6. c2tt6d0
/iscsi/disk@0000iqn.2006-01.com.openfilerrac1.asm30001,0Specify disk (enter its number):
^c
f) Display information about the iSCSI initiator.
# iscsiadm list initiator-node
Initiator node name: iqn.1986-03.com.sun:01:0003ba4d233b.425c293c
Initiator node alias: zzr1200
Login Parameters (Default/Configured):
Header Digest: NONE/-
Data Digest: NONE/-
Authentication Type: NONE
RADIUS Server: NONE
RADIUS access: unknown
g) Display information about which discovery methods are in use.
For example:
# Iscsiadm list discovery
Discovery:
Static: enabled
Send Targets: disabled
#iscsiadm list discovery-address -v 14.0.0.5
Discovery Address: 10.0.0.1:3260
Target name: eui.210000203787dfc0
Target address: 14.0.0.5:11824
Target name: eui.210000203787e07b
Target address: 14.0.0.5:11824
# iscsiadm list initiator-node
Initiator node name: iqn.1986-03.com.sun:01:0003ba4d233b.425c293c
Initiator node alias: zzr1200
Login Parameters (Default/Configured):
Header Digest: NONE/-
Data Digest: NONE/-
Authentication Type: NONE
RADIUS Server: NONE
RADIUS access: unknown
g) Display information about which discovery methods are in use.
For example:
# Iscsiadm list discovery
Discovery:
Static: enabled
Send Targets: disabled
#iscsiadm list discovery-address -v 14.0.0.5
Discovery Address: 10.0.0.1:3260
Target name: eui.210000203787dfc0
Target address: 14.0.0.5:11824
Target name: eui.210000203787e07b
Target address: 14.0.0.5:11824
Once the above configuration is done, on one of the solaris boxes, run the format command and create the partitions for the iSCSI volumes as below.
Note that you should leave the first 3 cylinders and start from the 3 rd cylinder while creating the partitions, so as not to overwrite the partition table. And partitioning should be done only on one node.
CHANGING OWNERSHIP OF CRS AND DATABASE FILES
Then do the following on one node.
# chown root:dba /dev/rdsk/c2t1d0s1 (OCR)
# chmod 660 /dev/rdsk/c2t1d0s1# chown oracle:dba /dev/rdsk/c2t2d0s1 (VOT)
# chmod 660 /dev/rdsk/c2t2d0s1# chown oracle:dba /dev/rdsk/c2t3d0s1 (ASM FILES)
# chown oracle:dba /dev/rdsk/c2t4d0s1# chown oracle:dba /dev/rdsk/c2t5d0s1# chown oracle:dba /dev/rdsk/c2t6d0s1 ON BOTH SOLARIS BOXES
1. Create dba group
#groupadd dba
2. create oracle user with dba group
#useradd -u 500 -g dba oracle #passwd oracle
Configuring SSH and user equivalency on both solaris boxes
Login as oracle user and run the following in home directory.
$ mkdir ~/.ssh
$ chmod 700 ~/.ssh
$ chmod 700
$ /usr/bin/ssh-keygen -t rsa (accept the defaults and press enter)
$ /usr/bin/ssh-keygen -t dsa (accept the defaults and press enter)
Create a test file ‘test’ as oracle user with the following entries.
ssh sola1.micro.com date
ssh sola2.micro.com date
ssh sola1-prv.micro.com date
ssh sola2-prv.micro.com date
ssh sola1 date
ssh sola2 date
ssh sola1-prv date
ssh sola2-prv date
#chmod 777 test
#./test
When it is run for the first time, enter “yes”, and enter password for oracle user. The following message may appear.
The authenticity of host 'sola1 (xxx)' can't be established.RSA key fingerprint is ez:e7:f6:f4:f2:4f:8f:9z:79:85:62:20:90:92:z9.
Are you sure you want to continue connecting (yes/no)?
Run the script for the second time and it should display the dates without asking for the password. This completes the user equivalency. The purpose of doing this is, during the CRS and Database installation, OUI is run only from one node and it will install the software on the local node first and then on remote nodes using ssh.
In case, if the above script is asking for password, then check the permissions of .ssh directory. It should be 600.
(Optional)
On the node where the OUI will be invoked, as oracle user run the following commands
$ exec /usr/bin/ssh-agent $SHELL
$ /usr/bin/ssh-add
After this, if we run the “test”, it will not ask for the password.
CREATING DIRECTORIES FOR ORACLE AND CRS INSTALLATIONS
As root user,
# mkdir /crs
#mkdir /dbhome
#chown –R oracle:dba crs dbhome
Setting Kernel values
In /etc/system, enter the following values on both the solaris boxes and reboot.
set noexec_user_stack=1
set semsys:seminfo_semmni=100
set semsys:seminfo_semmns=1024
set semsys:seminfo_semmsl=256
set semsys:seminfo_semvmx=32767
set shmsys:shminfo_shmmax=4294967295
set shmsys:shminfo_shmmin=1
set shmsys:shminfo_shmmni=100
set shmsys:shminfo_shmseg=10
(copy the project add oracle and shared memory of 4GB etc)
INSTALLATING ORACLE CLUSTERWARE
Oracle RAC installation is typically initiated from any one of the RAC nodes and the software is installed on the remaining nodes remotely using ssh/scp. The following should be done from any ONE of the RAC nodes. (For example, sola1).
On SOLA1 INSTALLING CLUSTERWARE
As root user, run #xhost +
login as oracle user
#DISPLAY=:0.0
#export DISPLAY
check if ssh is working fine by running “test” script created above. It should run successfully without asking for password.
#cd /clusterware
#./runInstaller
Then do the following on one node.
# chown root:dba /dev/rdsk/c2t1d0s1 (OCR)
# chmod 660 /dev/rdsk/c2t1d0s1# chown oracle:dba /dev/rdsk/c2t2d0s1 (VOT)
# chmod 660 /dev/rdsk/c2t2d0s1# chown oracle:dba /dev/rdsk/c2t3d0s1 (ASM FILES)
# chown oracle:dba /dev/rdsk/c2t4d0s1# chown oracle:dba /dev/rdsk/c2t5d0s1# chown oracle:dba /dev/rdsk/c2t6d0s1 ON BOTH SOLARIS BOXES
1. Create dba group
#groupadd dba
2. create oracle user with dba group
#useradd -u 500 -g dba oracle #passwd oracle
Configuring SSH and user equivalency on both solaris boxes
Login as oracle user and run the following in home directory.
$ mkdir ~/.ssh
$ chmod 700 ~/.ssh
$ chmod 700
$ /usr/bin/ssh-keygen -t rsa (accept the defaults and press enter)
$ /usr/bin/ssh-keygen -t dsa (accept the defaults and press enter)
Create a test file ‘test’ as oracle user with the following entries.
ssh sola1.micro.com date
ssh sola2.micro.com date
ssh sola1-prv.micro.com date
ssh sola2-prv.micro.com date
ssh sola1 date
ssh sola2 date
ssh sola1-prv date
ssh sola2-prv date
#chmod 777 test
#./test
When it is run for the first time, enter “yes”, and enter password for oracle user. The following message may appear.
The authenticity of host 'sola1 (xxx)' can't be established.RSA key fingerprint is ez:e7:f6:f4:f2:4f:8f:9z:79:85:62:20:90:92:z9.
Are you sure you want to continue connecting (yes/no)?
Run the script for the second time and it should display the dates without asking for the password. This completes the user equivalency. The purpose of doing this is, during the CRS and Database installation, OUI is run only from one node and it will install the software on the local node first and then on remote nodes using ssh.
In case, if the above script is asking for password, then check the permissions of .ssh directory. It should be 600.
(Optional)
On the node where the OUI will be invoked, as oracle user run the following commands
$ exec /usr/bin/ssh-agent $SHELL
$ /usr/bin/ssh-add
After this, if we run the “test”, it will not ask for the password.
CREATING DIRECTORIES FOR ORACLE AND CRS INSTALLATIONS
As root user,
# mkdir /crs
#mkdir /dbhome
#chown –R oracle:dba crs dbhome
Setting Kernel values
In /etc/system, enter the following values on both the solaris boxes and reboot.
set noexec_user_stack=1
set semsys:seminfo_semmni=100
set semsys:seminfo_semmns=1024
set semsys:seminfo_semmsl=256
set semsys:seminfo_semvmx=32767
set shmsys:shminfo_shmmax=4294967295
set shmsys:shminfo_shmmin=1
set shmsys:shminfo_shmmni=100
set shmsys:shminfo_shmseg=10
(copy the project add oracle and shared memory of 4GB etc)
INSTALLATING ORACLE CLUSTERWARE
Oracle RAC installation is typically initiated from any one of the RAC nodes and the software is installed on the remaining nodes remotely using ssh/scp. The following should be done from any ONE of the RAC nodes. (For example, sola1).
On SOLA1 INSTALLING CLUSTERWARE
As root user, run #xhost +
login as oracle user
#DISPLAY=:0.0
#export DISPLAY
check if ssh is working fine by running “test” script created above. It should run successfully without asking for password.
#cd /clusterware
#./runInstaller
(continued...)
Friday, March 9, 2007
Subscribe to:
Posts (Atom)