I encountered the below errors when i am trying to start the RAC DB/instances.
The Cluster/CRS is up but could not bring up the RAC DB instances.
Environment: 2 Node RAC (12cr2) on Linux 7.3
[grid@Server-666 ~]$ /oragrid/app/grid_home/bin/srvctl start database -db rac
PRCR-1079 : Failed to start resource ora.rac.db
CRS-5017: The resource action "ora.rac.db start" encountered the following error:
ORA-01078: failure in processing system parameters
LRM-00109: could not open parameter file '/ora00/app/oracle/product/12.2/db_1/dbs/initrac2.ora'
. For details refer to "(:CLSN00107:)" in "/oragrid/app/grid_base/diag/crs/server-777/crs/trace/crsd_oraagent_oracle.trc".
CRS-2674: Start of 'ora.rac.db' on 'server-777' failed
CRS-5017: The resource action "ora.rac.db start" encountered the following error:
ORA-01078: failure in processing system parameters
LRM-00109: could not open parameter file '/ora00/app/oracle/product/12.2/db_1/dbs/initrac1.ora'
. For details refer to "(:CLSN00107:)" in "/oragrid/app/grid_base/diag/crs/server-666/crs/trace/crsd_oraagent_oracle.trc".
CRS-2674: Start of 'ora.rac.db' on 'server-666' failed
CRS-2632: There are no more servers to try to place resource 'ora.rac.db' on that would satisfy its placement policy
================================================================================
Attempt to startup RAC DB instance 1 (rac1) fails … even if i point it to the default parameter file stored in ASM
1)
SQL> startup
ORA-01078: failure in processing system parameters
LRM-00109: could not open parameter file '/ora00/app/oracle/product/12.2/db_1/dbs/initrac1.ora'
2)
echo "SPFILE='+DATA/RAC/PARAMETERFILE/spfile.272.1014397511'" >
SQL> startup
ORA-01078: failure in processing system parameters
ORA-01565: error in identifying file '+DATA/RAC/PARAMETERFILE/spfile.272.1014397511'
ORA-17503: ksfdopn:10 Failed to open file +DATA/RAC/PARAMETERFILE/spfile.272.1014397511
ORA-12547: TNS:lost contact
ORA-12547: TNS:lost contact
SQL>
================================================================================
[grid@SERVER-666 ~]$ srvctl config database -d rac
Database unique name: RAC
Database name: RAC
Oracle home: /ora00/app/oracle/product/12.2/db_1
Oracle user: oracle
Spfile: +DATA/RAC/PARAMETERFILE/spfile.272.1014397511
Password file: +DATA/RAC/PASSWORD/pwdrac.256.1014397199
Domain: xxxx.xxxx.com
Start options: open
Stop options: immediate
Database role: PRIMARY
Management policy: AUTOMATIC
Server pools:
Disk Groups: DATA,UNDO_DISK,LOB_DISK
Mount point paths:
Services: aptra
Type: RAC
Start concurrency:
Stop concurrency:
OSDBA group: dba
OSOPER group: oper
Database instances: RAC1,RAC2
Configured nodes: server-666,server-777
CSS critical: no
CPU count: 0
Memory target: 0
Maximum memory: 0
Default network number for database services:
Database is administrator managed
================================================================================
===> Attempt to start RAC service fail.
[grid@server-666 ~]$ srvctl start service -d rac -s aptra
PRCD-1133 : failed to start services aptra for database rac
PRCR-1095 : Failed to start resources using filter ((NAME == ora.rac.aptra.svc) AND (TYPE == ora.service.type))
CRS-5017: The resource action "ora.rac.db start" encountered the following error:
ORA-01078: failure in processing system parameters
LRM-00109: could not open parameter file '/ora00/app/oracle/product/12.2/db_1/dbs/initrac2.ora'
. For details refer to "(:CLSN00107:)" in "/oragrid/app/grid_base/diag/crs/server-777/crs/trace/crsd_oraagent_oracle.trc".
CRS-2674: Start of 'ora.rac.db' on 'server-777' failed
CRS-5017: The resource action "ora.rac.db start" encountered the following error:
ORA-01078: failure in processing system parameters
ORA-01565: error in identifying file '+DATA/rac/PARAMETERFILE/spfile.272.1014397511'
ORA-17503: ksfdopn:10 Failed to open file +DATA/rac/PARAMETERFILE/spfile.272.1014397511
ORA-12547: TNS:lost contact
ORA-12547: TNS:lost contact
. For details refer to "(:CLSN00107:)" in "/oragrid/app/grid_base/diag/crs/server-666/crs/trace/crsd_oraagent_oracle.trc".
CRS-2674: Start of 'ora.rac.db' on 'server-666' failed
CRS-2632: There are no more servers to try to place resource 'ora.rac.aptra.svc' on that would satisfy its placement policy
================================================================================
===> Crs and ASM are up,
[grid@SERVER-666 ~]$ crsctl check cluster
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
[grid@SERVER-666 ~]$ crsctl check cluster -all
server-666:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
server-777:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
[grid@SERVER-666~]$ srvctl config scan
SCAN name: aptra-cluster-scan, Network: 1
Subnet IPv4: 192.168.119.0/255.255.255.0/ens224, static
Subnet IPv6:
SCAN 1 IPv4 VIP: 192.168.119.16
SCAN VIP is enabled.
SCAN VIP is individually enabled on nodes:
SCAN VIP is individually disabled on nodes:
SCAN 2 IPv4 VIP: 192.168.119.18
SCAN VIP is enabled.
SCAN VIP is individually enabled on nodes:
SCAN VIP is individually disabled on nodes:
SCAN 3 IPv4 VIP: 192.168.119.17
SCAN VIP is enabled.
SCAN VIP is individually enabled on nodes:
SCAN VIP is individually disabled on nodes:
[root@SERVER-666 ~]# /oragrid/app/grid_home/bin/srvctl status asm
ASM is running on SERVER-666,SERVER-777
Name Target State Server State details
Local Resources
ora.ASMNET1LSNR_ASM.lsnr
ONLINE ONLINE SERVER-666 STABLE
ONLINE ONLINE SERVER-777 STABLE
ora.CRS.dg
ONLINE ONLINE SERVER-666 STABLE
ONLINE ONLINE SERVER-777 STABLE
ora.DATA.dg
ONLINE ONLINE SERVER-666 STABLE
ONLINE ONLINE SERVER-777 STABLE
ora.LISTENER.lsnr
ONLINE ONLINE SERVER-666 STABLE
ONLINE ONLINE SERVER-777 STABLE
ora.LOB_DISK.dg
ONLINE ONLINE SERVER-666 STABLE
ONLINE ONLINE SERVER-777 STABLE
ora.MGMT.dg
ONLINE ONLINE SERVER-666 STABLE
ONLINE ONLINE SERVER-777 STABLE
ora.REDO_A.dg
ONLINE ONLINE SERVER-666 STABLE
ONLINE ONLINE SERVER-777 STABLE
ora.REDO_B.dg
ONLINE ONLINE SERVER-666 STABLE
ONLINE ONLINE SERVER-777 STABLE
ora.UNDO_DISK.dg
ONLINE ONLINE SERVER-666 STABLE
ONLINE ONLINE SERVER-777 STABLE
ora.chad
ONLINE ONLINE SERVER-666 STABLE
ONLINE ONLINE SERVER-777 STABLE
ora.net1.network
ONLINE ONLINE SERVER-666 STABLE
ONLINE ONLINE SERVER-777 STABLE
ora.ons
ONLINE ONLINE SERVER-666 STABLE
ONLINE ONLINE SERVER-777 STABLE
Cluster Resources
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE SERVER-666 STABLE
ora.LISTENER_SCAN2.lsnr
1 ONLINE ONLINE SERVER-777 STABLE
ora.LISTENER_SCAN3.lsnr
1 ONLINE ONLINE SERVER-777 STABLE
ora.MGMTLSNR
1 ONLINE ONLINE SERVER-777 169.254.240.140 10.1
0.1.3,STABLE
ora.asm
1 ONLINE ONLINE SERVER-666 Started,STABLE
2 ONLINE ONLINE SERVER-777 Started,STABLE
3 OFFLINE OFFLINE STABLE
ora.cvu
1 ONLINE ONLINE SERVER-777 STABLE
ora.mgmtdb
1 ONLINE ONLINE SERVER-777 Open,STABLE
ora.rac.aptra.svc
1 ONLINE OFFLINE STABLE
2 OFFLINE OFFLINE STABLE
ora.rac.db
1 ONLINE OFFLINE STABLE
2 ONLINE OFFLINE STABLE
ora.qosmserver
1 ONLINE ONLINE SERVER-777 STABLE
ora.scan1.vip
1 ONLINE ONLINE SERVER-666 STABLE
ora.scan2.vip
1 ONLINE ONLINE SERVER-777 STABLE
ora.scan3.vip
1 ONLINE ONLINE SERVER-777 STABLE
ora.SERVER-666.vip
1 ONLINE ONLINE SERVER-666 STABLE
ora.SERVER-777.vip
1 ONLINE ONLINE SERVER-777 STABLE
Best Answer
I figured it out, its due to file permission issue under $GRID_HOME
ls -l $GRID_HOME/bin/oracle
chmod 6751 $GRID_HOME/bin/oracle
Once permissions are fixed, try restarting db services.