Oracle 12 startup issue: disk full

oraclerman

Ok, here's my situation: I have the following set up:

  • Oracle 12c database
  • Data storage is through ASM on a raw disk partition
  • Server is running on a Windows 2012 server
    • Database was in archivelog mode

When I connect to Oracle through SQL Plus, I get a 'connected to an idle instance' message. When I attempt to run 'startup', I get the 'Oracle instance started/database mounted' message, but then I get an 'ORA-03113: End-of-file on communication channel' message.

I took a look through 'diag/rdbms/orcl/orcl/trace/alert_orcl.txt'. (I think that's the file I should be using.. I've never really had to deal with this type of error before.) The file has the following

sksasmowrt WriteConsole error 6 Fri Oct 07 11:21:40 2016 Unable to
create archive log file '+DATA' Fri Oct 07 11:21:40 2016 Errors in
file C:\ORACLE\diag\rdbms\orcl\orcl\trace\orcl_arc0_1512.trc:
ORA-19816: WARNING: Files may exist in db_recovery_file_dest that are
not known to database. ORA-17502: ksfdcre:4 Failed to create file
+DATA ORA-15041: diskgroup "DATA" space exhausted
************************************************************* WARNING: A file of type ARCHIVED LOG may exist in db_recovery_file_dest that is
not known to the database. Use the RMAN command CATALOG RECOVERY AREA
to re-catalog any such files. If files cannot be cataloged, then
manually delete them using OS command. This is most likely the result
of a crash during file creation.
************************************************************* Fri Oct 07 11:21:40 2016 Errors in file
C:\ORACLE\diag\rdbms\orcl\orcl\trace\orcl_arc0_1512.trc: ORA-19504:
failed to create file "+DATA" ORA-17502: ksfdcre:4 Failed to create
file +DATA ORA-15041: diskgroup "DATA" space exhausted ARC0: Error
19504 Creating archive log file to '+DATA' Fri Oct 07 11:21:40 2016
Unable to create archive log file '+DATA' Fri Oct 07 11:21:40 2016
Errors in file C:\ORACLE\diag\rdbms\orcl\orcl\trace\orcl_ora_3564.trc:
ORA-19816: WARNING: Files may exist in db_recovery_file_dest that are
not known to database. ORA-17502: ksfdcre:4 Failed to create file
+DATA ORA-15041: diskgroup "DATA" space exhausted
************************************************************* WARNING: A file of type ARCHIVED LOG may exist in db_recovery_file_dest that is
not known to the database. Use the RMAN command CATALOG RECOVERY AREA
to re-catalog any such files. If files cannot be cataloged, then
manually delete them using OS command. This is most likely the result
of a crash during file creation.
************************************************************* Fri Oct 07 11:21:40 2016 Errors in file
C:\ORACLE\diag\rdbms\orcl\orcl\trace\orcl_ora_3564.trc: ORA-19504:
failed to create file "+DATA" ORA-17502: ksfdcre:4 Failed to create
file +DATA ORA-15041: diskgroup "DATA" space exhausted ARCH: Error
19504 Creating archive log file to '+DATA' Fri Oct 07 11:21:40 2016
Errors in file C:\ORACLE\diag\rdbms\orcl\orcl\trace\orcl_ora_3564.trc:
ORA-16038: log 2 sequence# 188 cannot be archived ORA-19504: failed to
create file "" ORA-00312: online log 2 thread 1:
'+DATA/ORCL/ONLINELOG/group_2.265.921320001' ORA-00312: online log 2
thread 1: '+DATA/ORCL/ONLINELOG/group_2.266.921320001' USER (ospid:
3564): terminating the instance due to error 16038 Fri Oct 07 11:21:41
2016 System state dump requested by (instance=1, osid=3564),
summary=[abnormal instance termination]. System State dumped to trace
file
C:\ORACLE\diag\rdbms\orcl\orcl\trace\orcl_diag_712_20161007112141.trc
Fri Oct 07 11:21:42 2016 License high water mark = 2 Fri Oct 07
11:21:49 2016 Instance terminated by USER, pid = 3564

So, it LOOKS like there is an issue with some sort of unused files clogging up the storage area. (Unless I'm totally misreading things.)

The trace file above suggests using RMAN to catalog the file or the OS to delete the file, but I'm not very familiar with RMAN, nor how to handle RAW partitions. (Normally I use NTFS but I'm trying something new here.) Searching on line has found similar problems but they seem to be either on Linux, or using NTFS

Anyone have any suggestions on how to proceed?

I've tried the following so far:

RMAN> connect target sys/(the password)
connected to target database (not started)
RMAN> catalog recovery area
using target database control file instead of recovery catalog
RMAN-06403: could not obtain a fully authorized session
ORA-01034: ORACLE not available
ORA-27101: shared memory realm does not exist

And once things are working properly, does anyone have any suggestions on how to prevent the issue from happening in the future?
Thanks in advance.

Best Answer

The database in the above will not open. Start the database in MOUNT mode, and clean up the archivelog area.

rman
RMAN> connect target sys/(the password)
RMAN> startup mount
RMAN> catalog recovery area;

Now you could clean up archivelogs, for example:

RMAN> delete archivelog all;

Finally, open the database:

RMAN> alter database open;

To prevent this from happening in the future, schedule a regular database backup + archivelog backup that deletes archivelogs. If you do not need backups and archivelogs, just disable archivelog mode while in MOUNT stage:

alter database noarchivelog;