Problem summary
When browsing to the Oracle 12c EM Express webinterface, the page times-out when using a URL like:
- http(s)://hostname:5500/em
- http(s)://hostname.domain:5500/em
When using a url containing the IP address, or localhost, the page loads, but very very slowly (up to 5 minutes).
- http(s)://ip-address/em
- http(s)://localhost/em
When using the hostname, the listener log shows errors (multiple of these):
03-DEC-2015 11:25:33 * http * (ADDRESS=(PROTOCOL=tcps)(HOST=<ip address>%14)(PORT=52925)) * handoff * http * 12518
TNS-12518: TNS:listener could not hand off client connection
TNS-12560: TNS:protocol adapter error
When using the IP address or localhost, the listener log shows no errors:
03-DEC-2015 11:23:39 * http * (ADDRESS=(PROTOCOL=tcps)(HOST=<ip address>)(PORT=52902)) * handoff * http * 0
This problem shows up all over the web (see references in footer), and nobody seems to be able to find the cause of this. I've tried -every- suggestion i could find, and listed the results below. I'll let this great community have a crack at it, and then it's off to Oracle support…
Platform information
- Windows Server 2012 R2 x64
- Oracle 12.1.0.2.0
- A multitenant container database (CDB) with two Pluggable databases (PDB)
- Automatic Storage Management (ASM) is not being used
- Performing tests locally on the server, with firewall completely disabled
Tests already performed
emctl
A lot of people would suggest checking "emctl" out of habit. This does not exist for 12c, as it uses EM Express and not Database control.
http/https
I've set up, and tested this for http and https. No difference.
Browser
I've tested this in vanilla installations of IE 11, chrome and firefox. No add-ons enabled.
Database access
Both the CDB and PDB's can be accessed and queries through SQLPlus
Telnet
Opening telnet connections to the ports (e.g. 5500) works.
Alert log
The Oracle alert log does not contain any error messages.
My hypotheses
- Something related to hostname lookup, since it works -slighty- when using IP address instead of hostname.
- Something related to the certificate, although the same problem shows up for http.
- Something related to the configuration of the XDB or listener service.
- Something related to the domain part of the hostname being included or not. As this is added in most of the configuration files, but maybe not everywhere?
Detailed platform information
Note: In these snippets, i've always replaced the actual hostname with "HOSTNAME" and the actual domain with "domain". If "HOSTNAME" is used, it means it was not trailed by the domain name. The case represents the actual case the hostname is displayed in.
SYS_CONTEXT('USERENV','SERVER_HOST')
hostname
hostname variable in CMD
HOSTNAME
SYS_CONTEXT('USERENV','DB_DOMAIN')
domain
dbms_xdb_config.gethttpsport()
5500
dbms_xdb_config.gethttpport()
2200
"lsnrctl status" output
LSNRCTL for 64-bit Windows: Version 12.1.0.2.0 - Production on 03-DEC-2015 15:33:12
Copyright (c) 1991, 2014, Oracle. All rights reserved.
Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=hostname.domain)(PORT=1521)))
STATUS of the LISTENER
------------------------
Alias listener
Version TNSLSNR for 64-bit Windows: Version 12.1.0.2.0 - Production
Start Date 03-DEC-2015 12:27:51
Uptime 0 days 3 hr. 5 min. 21 sec
Trace Level off
Security ON: Local OS Authentication
SNMP OFF
Listener Parameter File C:\ORACLE\HOME\network\admin\listener.ora
Listener Log File C:\ORACLE\diag\tnslsnr\HOSTNAME\listener\alert\log.xml
Listening Endpoints Summary...
(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=HOSTNAME.domain)(PORT=1521)))
(DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(PIPENAME=\\.\pipe\EXTPROC1521ipc)))
(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=HOSTNAME.domain)(PORT=2200))(Presentation=HTTP)(Session=RAW))
(DESCRIPTION=(ADDRESS=(PROTOCOL=tcps)(HOST=HOSTNAME.domain)(PORT=5501))(Security=(my_wallet_directory=C:\ORACLE\admin\ecoomdb\xdb_wallet))(Presentation=HTTP)(Session=RAW))
(DESCRIPTION=(ADDRESS=(PROTOCOL=tcps)(HOST=HOSTNAME.domain)(PORT=5500))(Security=(my_wallet_directory=C:\ORACLE\admin\ecoomdb\xdb_wallet))(Presentation=HTTP)(Session=RAW))
Services Summary...
Service "CLRExtProc" has 1 instance(s).
Instance "CLRExtProc", status UNKNOWN, has 1 handler(s) for this service...
Service "bibliodb.domain" has 1 instance(s).
Instance "ecoomdb", status READY, has 1 handler(s) for this service...
Service "ecoomdb.domain" has 1 instance(s).
Instance "ecoomdb", status READY, has 1 handler(s) for this service...
Service "ecoomdbXDB.domain" has 1 instance(s).
Instance "ecoomdb", status READY, has 1 handler(s) for this service...
Service "technodb.domain" has 1 instance(s).
Instance "ecoomdb", status READY, has 1 handler(s) for this service...
The command completed successfully
"lsnrctl services" output
LSNRCTL for 64-bit Windows: Version 12.1.0.2.0 - Production on 03-DEC-2015 15:34:22
Copyright (c) 1991, 2014, Oracle. All rights reserved.
Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=HOSTNAME.domain)(PORT=1521)))
Services Summary...
Service "CLRExtProc" has 1 instance(s).
Instance "CLRExtProc", status UNKNOWN, has 1 handler(s) for this service...
Handler(s):
"DEDICATED" established:0 refused:0
LOCAL SERVER
Service "bibliodb.domain" has 1 instance(s).
Instance "ecoomdb", status READY, has 1 handler(s) for this service...
Handler(s):
"DEDICATED" established:9 refused:0 state:ready
LOCAL SERVER
Service "ecoomdb.domain" has 1 instance(s).
Instance "ecoomdb", status READY, has 1 handler(s) for this service...
Handler(s):
"DEDICATED" established:9 refused:0 state:ready
LOCAL SERVER
Service "ecoomdbXDB.domain" has 1 instance(s).
Instance "ecoomdb", status READY, has 1 handler(s) for this service...
Handler(s):
"D000" established:8 refused:21 current:0 max:1022 state:ready
DISPATCHER <machine: HOSTNAME, pid: 4040>
(ADDRESS=(PROTOCOL=tcp)(HOST=HOSTNAME.domain)(PORT=49179))
Service "technodb.domain" has 1 instance(s).
Instance "ecoomdb", status READY, has 1 handler(s) for this service...
Handler(s):
"DEDICATED" established:9 refused:0 state:ready
LOCAL SERVER
The command completed successfully
listener.ora
SID_LIST_LISTENER =
(SID_LIST =
(SID_DESC =
(SID_NAME = CLRExtProc)
(ORACLE_HOME = C:\ORACLE\HOME)
(PROGRAM = extproc)
(ENVS = "EXTPROC_DLLS=ONLY:C:\ORACLE\HOME\bin\oraclr12.dll")
)
)
LISTENER =
(DESCRIPTION_LIST =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = HOSTNAME.domain)(PORT = 1521))
(ADDRESS = (PROTOCOL = IPC)(KEY = EXTPROC1521))
)
)
init.ora
(Trimmed, also contains other settings…)
db_domain=domain
db_name="ecoomdb"
dispatchers="(PROTOCOL=TCP) (SERVICE=ecoomdbXDB)"
local_listener=LISTENER*
netstat -a
(relevant selection)
TCP 0.0.0.0:1521 ECONECOOMDB1:0 LISTENING
TCP 0.0.0.0:5500 ECONECOOMDB1:0 LISTENING
TCP 0.0.0.0:5501 ECONECOOMDB1:0 LISTENING
TCP [::]:5500 ECONECOOMDB1:0 LISTENING
TCP [::]:5501 ECONECOOMDB1:0 LISTENING
windows hosts file
(this file is empty)
Environment variables
- PATH contains C:\ORACLE\HOME\bin
- ORACLE_HOME = C:\ORACLE\HOME
- ORACLE_SID = ecoomdb
Registry
HKEY_LOCAL_MACHINE\SOFTWARE\ORACLE\KEY_OraDB12Home1
- ORACLE_HOME = C:\ORACLE\HOME
- ORACLE_SID = ecoomdb
Related and duplicate (unanswered) questions/documents
Online resources
Stackexchange questions
- Unable to open/connect to Oracle 12c Enterprise Manager Express
- "Unable to display page" with Oracle 12c Enterprise Manager Database Express
- Start Oracle 12c enterprise manager in Windows 7
- How to run ORACLE 12C EM?
- Oracle 12c Express Enterprise Manager on Centos 6.5 is not reachable
Reddit threads
-
https://www.reddit.com/r/oracle/comments/2pq7wz/oracle_12c_express_enterprise_manager/
-
https://www.reddit.com/r/oracle/comments/29npk2/having_trouble_getting_enterprise_manager_express/
Oracle community
- https://community.oracle.com/thread/3595774?start=0&tstart=0
- https://community.oracle.com/thread/2590962?start=0&tstart=0
- https://community.oracle.com/thread/3730316?start=0&tstart=0
- https://community.oracle.com/thread/3739356?start=0&tstart=0
- https://community.oracle.com/thread/3682977?start=0&tstart=0
Best Answer
After contacting Oracle support, I've -kinda- been able to figure out what was going on. This server has 4 network adaptors, 1 adaptor connected to the network, and 3 unconfigured adaptors connected to an iSCSI storage. The problem seems to be caused by the web-client, or EM server (not sure here) trying to send the webpage request to the wrong adaptor.
For anyone facing the same issue, you can check this by browsing to the URL (using the hostname.domain) and then checking the listener.log file (-oracle_base-/diag/tnslsnr/-instance-/listener/trace/listener.log). For me, in the error message a different IP address than the servers main network adaptors IP address was shown. The IP address of one of the network adaptors connected to the iSCSI where shown. When omitting the IP addresses in my original post, I didn't spot this IP address difference.
When disabling the additional network adaptors, everything works fine.
Additionally, when going to "chrome://net-internals/#dns" in chrome, the IP addresses of all four network adaptors showed up for my hostname.
Apparently, this problem -does not- happen, when browsing to the webpage from another host. Also, nslookup returns only the correct IP address. So it seems this is not a DNS issue, but a local issue. I think this is not a full answer to the problem, as disabling network adaptors can't be a final solution, but I hope anyone facing this issue in the future has "more to go on" now...
Where does the discrepancy between IP address returned by the DNS server, and the IP addresses used by web browsers come from?
I've also made some additional changes that might have influenced the end-solution, so for completeness I'll include them. Not sure if relevant...: