955
Oralce RAC On ECS（共享雲盤）部署方法
1 概述
描述ORACLE RAC架構使用阿裏雲ECS（IO優化實例、獨享型 sn2）、SSD共享雲盤配置Oracle RAC方法和步驟。
2 目的
基於阿裏雲ECS共享SSD磁盤 完成ORACLE RAC的部署。
3 環境準備
環境準備步驟如下：
申請雲主機，包括2個RAC節點、supernode節點（同時用作壓測機）
購買SSD共享雲盤3塊
3.1 網絡拓撲圖
壓測拓撲
RAC ON ECS 拓撲
3.2 RAC規劃
3.3 修改主機名
vi /etc/sysconfig/network
節點1
NETWORKING=yes
HOSTNAME=rac1
NETWORKING_IPV6=no
PEERNTP=no
GATEWAY=47.93.139.247
NOZEROCONF=yes
節點2
NETWORKING=yes
HOSTNAME=rac2
NETWORKING_IPV6=no
PEERNTP=no
GATEWAY=47.93.139.247
NOZEROCONF=yes
3.4 安裝n2n軟件
在supernode節點安裝ntop軟件
cd /usr/src
svn co https://svn.ntop.org/svn/ntop/trunk/n2n
cd n2n/n2n_v2
make
make PREFIX=/opt/n2n install
安裝完畢後啟動supernode服務
/opt/n2n/sbin/supernode -l 65530
/opt/n2n/sbin/edge -d edge0 -a 10.10.10.111 -s 255.255.255.0 -c dtstack -k dtstack -l 10.30.51.64:65530 -E –r #用作壓測使用
在RAC節點安裝ntop軟件（計劃加入vpn網絡的節點都需要安裝ntop）
scp /opt/n2n 10.30.92.237:/opt
運行下麵的命令，並寫入到/etc/rc.local
節點1
/opt/n2n/sbin/edge -d edge0 -a 10.10.10.101 -s 255.255.255.0 -c dtstack -k dtstack -l 10.30.51.64:65530 -E -r
/opt/n2n/sbin/edge -d edge1 -a 192.168.100.101 -s 255.255.255.0 -c dtstack -k dtstack -l 10.30.51.64:65530 -E -r
節點2
/opt/n2n/sbin/edge -d edge0 -a 10.10.10.102 -s 255.255.255.0 -c dtstack -k dtstack -l 10.30.51.64:65530 -E -r
/opt/n2n/sbin/edge -d edge1 -a 192.168.100.102 -s 255.255.255.0 -c dtstack -k dtstack -l 10.30.51.64:65530 -E -r
3.5 修改/etc/hosts
節點1
10.10.10.101 rac1
10.10.10.102 rac2
10.10.10.103 rac1-vip
10.10.10.104 rac2-vip
192.168.100.101 rac1-priv
192.168.100.102 rac2-priv
10.10.10.105 scan-ip
10.30.92.33 rac1
節點2
10.10.10.101 rac1
10.10.10.102 rac2
10.10.10.103 rac1-vip
10.10.10.104 rac2-vip
192.168.100.101 rac1-priv
192.168.100.102 rac2-priv
10.10.10.105 scan-ip
10.30.92.237 rac2
3.6 修改內核參數
vi /etc/sysctl.conf
fs.aio-max-nr = 1048576
fs.file-max = 6815744
kernel.shmall =7864320 #內存大小/4096分頁大小
kernel.shmmax = 52451655680 #內存大小
kernel.shmmni = 4096
kernel.sem = 250 32000 100 128
net.ipv4.ip_local_port_range = 9000 65500
net.core.rmem_default = 262144
net.core.rmem_max = 4194304
net.core.wmem_default = 262144
net.core.wmem_max = 1048586
vm.nr_hugepages = sga/pagesize #開啟hugepage，oracle鎖定內存防止換出
sysctl –p
3.7 配置NTP
oracle rac必須進行時鍾同步，如果沒有時鍾同步，按照下麵的方式配置，讓oracle自身來解決
/sbin/service ntpd stop
chkconfig ntpd off
mv /etc/ntp.conf /etc/ntp.conf.org
3.8 關閉防火牆
chkconfig --list iptables
chkconfig iptables off
chkconfig --list iptables
service iptables stop
service network restart
關閉selinux
/etc/selinux/config
disable
3.9 修改用戶限製
vi /etc/security/limits.conf
oracle soft nofile 4096
oracle hard nofile 65536
oracle soft nproc 2047
oracle hard nproc 16384
oracle soft stack 10240
grid soft nofile 4096
grid hard nofile 65536
grid soft nproc 2047
grid hard nproc 16384
grid soft stack 10240
* soft memlock 18874368
* hard memlock 18874368
以上memlock<ram的大小 單位為k
3.10 /etc/pam.d/login
echo "session required pam_limits.so" >>/etc/pam.d/login
3.11 安裝OS軟件包
rpm -q binutils \
compat-libstdc++-33 \
elfutils-libelf \
elfutils-libelf-devel \
gcc \
gcc-c++ \
glibc \
glibc-common \
glibc-devel \
glibc-headers \
ksh \
libaio \
libaio-devel \
libgcc \
libstdc++ \
libstdc++-devel \
libXp \
make \
numactl-devel \
sysstat \
unixODBC \
unixODBC-devel \
compat-libcap1.x86_64 \
libcap.x86_64|grep not
一般要安裝下麵的軟件包
yum install -y compat-libstdc++-33
yum install -y elfutils-libelf-devel
yum install -y gcc-c++
yum install -y ksh
yum install -y libaio-devel
yum install -y libstdc++-devel
yum install -y libXp
yum install -y numactl-devel
yum install -y unixODBC
yum install -y unixODBC-devel
yum install -y compat-libcap1.x86_64
3.12 格式化共享磁盤
每個磁盤分區成2個，2GB磁盤+剩餘空間
Disk /dev/vdb: 322.1 GB, 322122547200 bytes
Disk /dev/vdc: 322.1 GB, 322122547200 bytes
Disk /dev/vdd: 322.1 GB, 322122547200 bytes
3.13 安裝oracleasmlib軟件包
本次安裝使用了oracleasmlib管理asm磁盤
首先從linux-kernel安裝Kmod-oracleasm，這是linux源中自帶的，是linux支持oracleasm的軟件包
然後從oracle官網下載oracleasmlib和oracleasm-support
/usr/sbin/oracleasm configure -i #兩個節點
/etc/init.d/oracleasm enable #兩個節點
/etc/init.d/oracleasm start #兩個節點
/usr/sbin/oracleasm listdisks
/usr/sbin/oracleasm querydisks
/usr/sbin/oracleasm createdisk NAME /path
/usr/sbin/oracleasm scandisks #一個節點創建磁盤後，另一個節點掃描磁盤即可
3.14 準備ASM磁盤
本次創建的磁盤
/etc/init.d/oracleasm createdisk VOLCRS01 /dev/vdb1
/etc/init.d/oracleasm createdisk VOLCRS02 /dev/vdc1
/etc/init.d/oracleasm createdisk VOLCRS03 /dev/vdd1
/etc/init.d/oracleasm createdisk VOLDATA01 /dev/vdb2
/etc/init.d/oracleasm createdisk VOLDATA02 /dev/vdc2
/etc/init.d/oracleasm createdisk VOLDATA03 /dev/vdd2
3.15 創建grid、oracle用戶
/usr/sbin/groupadd -g 501 oinstall
/usr/sbin/groupadd -g 502 asmadmin
/usr/sbin/groupadd -g 503 dba
/usr/sbin/groupadd -g 504 oper
/usr/sbin/groupadd -g 505 asmdba
/usr/sbin/groupadd -g 506 asmoper
/usr/sbin/useradd -u 501 -g oinstall -G asmadmin,asmdba,asmoper,oper,dba grid
/usr/sbin/useradd -u 502 -g oinstall -G dba,asmdba,oper oracle
配置環境變量
Grid
Vi .bash_profile
umask 022
export ORACLE_BASE=/u01/app/grid_base
export ORACLE_HOME=/u01/app/grid_home
export ORACLE_SID=+ASM1 #第二個節點+ASM2
export PATH=/usr/sbin:$PATH
export PATH=$ORACLE_HOME/bin:$PATH
Oracle
umask 022
export ORACLE_BASE=/u01/app/oracle
export ORACLE_HOME=$ORACLE_BASE/product/11.2.0/db_1
export ORA_GRID_HOME=/u01/app/grid_home
export ORACLE_UNQNAME=orcl #第二個節點orcl2
export ORACLE_SID=orcl1
export PATH=/usr/sbin:$PATH
export PATH=$ORACLE_HOME/bin:$PATH
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$ORACLE_HOME/lib:/usr/lib
export CLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib
export NLS_LANG=AMERICAN_AMERICA.ZHS16GBK
export NLS_DATE_FORMAT="YYYY-MM-DD HH24:MI:SS"
export TNS_ADMIN=$ORACLE_HOME/network/admin
export LANG=en_US
3.16 創建安裝目錄
mkdir -p /u01/app/
mkdir –p /u01/app/grid_base
mkdir –p /u01/app/grid_home
chown -R grid:oinstall /u01/app/grid_base
chown -R grid:oinstall /u01/app/grid_home
chown -R oracle:oinstall /u01/app
chmod -R 775 /u01/app/
4 安裝GRID
4.1 準備軟件
軟件上傳至/u01/software
p13390677_112040_Linux-x86-64_2of7.zip
p13390677_112040_Linux-x86-64_1of7.zip
p13390677_112040_Linux-x86-64_3of7.zip
4.2 配置對等性
準備秘鑰
su - grid
mkdir ~/.ssh
ssh-keygen -t rsa
ssh-keygen -t dsa
su - oracle
mkdir ~/.ssh
ssh-keygen -t rsa
ssh-keygen -t dsa
在節點1執行
su - grid
cat ~/.ssh/id_rsa.pub>>./.ssh/authorized_keys --公鑰存在authorized_keys文件中,寫到本機
cat ~/.ssh/id_dsa.pub>>./.ssh/authorized_keys
ssh testOracle3Z-PRI cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys --第二個節點的公鑰寫到本機
ssh testOracle3Z-PRI cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
scp ~/.ssh/authorized_keys testOracle3Z-PRI:~/.ssh/authorized_keys
su - oracle
cat ~/.ssh/id_rsa.pub>>./.ssh/authorized_keys --公鑰存在authorized_keys文件中,寫到本機
cat ~/.ssh/id_dsa.pub>>./.ssh/authorized_keys
ssh testOracle3Z-PRI cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys --第二個節點的公鑰寫到本機
ssh testOracle3Z-PRI cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
scp ~/.ssh/authorized_keys testOracle3Z-PRI:~/.ssh/authorized_keys
驗證
ssh rac1 date (public網卡)
ssh rac2 date
ssh rac1-priv date (private網卡)
ssh rac2-priv date
4.3 安裝預檢查
su - grid
$cd grid的軟件目錄
$./runcluvfy.sh stage -pre crsinst -n rac1,rac2 -verbose
4.4 準備響應文件
ORACLE_HOSTNAME=rac1
INVENTORY_LOCATION=/u01/app/oraInventory
oracle.install.option=CRS_CONFIG
ORACLE_BASE=/u01/app/grid_base
ORACLE_HOME=/u01/app/grid_home
oracle.install.asm.OSDBA=asmdba
oracle.install.asm.OSOPER=asmoper
oracle.install.asm.OSASM=asmadmin
oracle.install.crs.config.gpnp.scanPort=1521
oracle.install.crs.config.clusterName=rac-cluster
oracle.install.crs.config.clusterNodes=rac1:rac1-vip,rac2:rac2-vip
oracle.install.crs.config.privateInterconnects=edge1:192.168.100.0:2,edge0:10.10.10.0:1 //1代表public，2代表private，3代表在群集中不使用該網卡
oracle.install.crs.config.storageOption=ASM_STORAGE
oracle.install.asm.SYSASMPassword=Grid123
oracle.install.asm.diskGroup.name=OCR
oracle.install.asm.diskGroup.redundancy=NORMAL
oracle.install.asm.diskGroup.AUSize=1
oracle.install.asm.diskGroup.disks=/dev/oracleasm/disks/VOLCRS01,/dev/oracleasm/disks/VOLCRS02,/dev/oracleasm/disks/VOLCRS03
oracle.install.asm.diskGroup.diskDiscoveryString=/dev/oracleasm/disks
oracle.install.asm.monitorPassword=Grid123
4.5 安裝grid
$./runInstaller -silent -responseFile /u01/software/grid/response/crs_install_test.rsp -ignoreSysPrereqs -ignorePrereq
等待一段時間後，會提示在兩個節點分別用root用戶執行：
/u01/grid/oraInventory/orainstRoot.sh
/u01/crs/11.2.0/root.sh
4.6 創建磁盤組
su – grid
$asmca
或者使用命令行
$ asmca -silent -createDiskGroup -sysAsmPassword Grid123 -diskString '/dev/oracleasm/disks/*' -diskGroupName DATA -diskList '/dev/oracleasm/disks/VOLDATA1' -redundancy NORMAL -compatible.asm 11.2 -compatible.rdbms 11.2
$ asmca -silent -addDisk -sysAsmPassword Grid123 -diskGroupName DATA -diskList '/dev/oracleasm/disks/VOLDATA2'
$ asmca -silent -addDisk -sysAsmPassword Grid123 -diskGroupName DATA -diskList '/dev/oracleasm/disks/VOLDATA3'
5 安裝Database
5.1 注冊產品列表
/u01/app/oraInventory/ContentsXMLinventory.xml
/u01/app/oraInventory/ContentsXML
5.2 準備響應文件
/u01/soft/oracle/database/response/db_install_test.rsp
oracle.install.option=INSTALL_DB_SWONLY
ORACLE_HOSTNAME=testOracle1Z.com
UNIX_GROUP_NAME=oinstall
INVENTORY_LOCATION=/u01/app/oraInventory
SELECTED_LANGUAGES=en
ORACLE_HOME=/u01/app/oracle/product/11.2.0/db_1
ORACLE_BASE=/u01/app/oracle
oracle.install.db.InstallEdition=EE
oracle.install.db.isCustomInstall=false
oracle.install.db.optionalComponents=oracle.rdbms.partitioning:11.2.0.4.0,oracle.oraolap:11.2.0.4.0,oracle.rdbms.dm:11.2.0.4.0,oracle.rdbms.dv:11.2.0.4.0,oracle.rdbms.lbac:11.2.0.4.0,oracle.r
dbms.rat:11.2.0.4.0oracle.install.db.DBA_GROUP=dba
oracle.install.db.OPER_GROUP=oinstall
oracle.install.db.CLUSTER_NODES=rac1,rac2
DECLINE_SECURITY_UPDATES=true
5.3 安裝
./runInstaller -silent -ignoreSysPrereqs -ignorePrereq -responseFile /u01/software/database/response/db_install_test.rsp
/u01/grid/oraInventory/orainstRoot.sh
/u01/oracle/11.2.0/root.sh
6 創建數據庫
創建數據庫可以使用dbca交互界麵，也可以使用靜默創建方式
6.1 準備響應文件
/u01/soft/oracle/database/response/dbca_orcl.rsp
GDBNAME = "orcl"
SID = "orcl"
NODELIST=rac1,rac2
SYSPASSWORD = "Oracle123"
SYSTEMPASSWORD = "Oracle123"
STORAGETYPE=ASM
DISKGROUPNAME=DATA
RECOVERYGROUPNAME=DATA
CHARACTERSET = "ZHS16GBK"
6.2 創建數據庫
dbca -silent -responseFile /u01/software/database/response/dbca_orcl.rsp
最後更新：2017-05-26 17:31:13