Oracle 12c Express Enterprise Manager webpage does not load

enterprise-manageroracleoracle-12coracle-enterprise-manager

Problem summary

When browsing to the Oracle 12c EM Express webinterface, the page times-out when using a URL like:

  • http(s)://hostname:5500/em
  • http(s)://hostname.domain:5500/em

When using a url containing the IP address, or localhost, the page loads, but very very slowly (up to 5 minutes).

  • http(s)://ip-address/em
  • http(s)://localhost/em

When using the hostname, the listener log shows errors (multiple of these):

03-DEC-2015 11:25:33 * http * (ADDRESS=(PROTOCOL=tcps)(HOST=<ip address>%14)(PORT=52925)) * handoff * http * 12518
TNS-12518: TNS:listener could not hand off client connection
TNS-12560: TNS:protocol adapter error

When using the IP address or localhost, the listener log shows no errors:

03-DEC-2015 11:23:39 * http * (ADDRESS=(PROTOCOL=tcps)(HOST=<ip address>)(PORT=52902)) * handoff * http * 0

This problem shows up all over the web (see references in footer), and nobody seems to be able to find the cause of this. I've tried -every- suggestion i could find, and listed the results below. I'll let this great community have a crack at it, and then it's off to Oracle support…

Platform information

  • Windows Server 2012 R2 x64
  • Oracle 12.1.0.2.0
  • A multitenant container database (CDB) with two Pluggable databases (PDB)
  • Automatic Storage Management (ASM) is not being used
  • Performing tests locally on the server, with firewall completely disabled

Tests already performed

emctl
A lot of people would suggest checking "emctl" out of habit. This does not exist for 12c, as it uses EM Express and not Database control.

http/https
I've set up, and tested this for http and https. No difference.

Browser
I've tested this in vanilla installations of IE 11, chrome and firefox. No add-ons enabled.

Database access
Both the CDB and PDB's can be accessed and queries through SQLPlus

Telnet
Opening telnet connections to the ports (e.g. 5500) works.

Alert log
The Oracle alert log does not contain any error messages.

My hypotheses

  • Something related to hostname lookup, since it works -slighty- when using IP address instead of hostname.
  • Something related to the certificate, although the same problem shows up for http.
  • Something related to the configuration of the XDB or listener service.
  • Something related to the domain part of the hostname being included or not. As this is added in most of the configuration files, but maybe not everywhere?

Detailed platform information

Note: In these snippets, i've always replaced the actual hostname with "HOSTNAME" and the actual domain with "domain". If "HOSTNAME" is used, it means it was not trailed by the domain name. The case represents the actual case the hostname is displayed in.

SYS_CONTEXT('USERENV','SERVER_HOST')
hostname

hostname variable in CMD
HOSTNAME

SYS_CONTEXT('USERENV','DB_DOMAIN')
domain

dbms_xdb_config.gethttpsport()
5500

dbms_xdb_config.gethttpport()
2200

"lsnrctl status" output

    LSNRCTL for 64-bit Windows: Version 12.1.0.2.0 - Production on 03-DEC-2015 15:33:12

    Copyright (c) 1991, 2014, Oracle.  All rights reserved.

    Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=hostname.domain)(PORT=1521)))
    STATUS of the LISTENER
    ------------------------
    Alias                     listener
    Version                   TNSLSNR for 64-bit Windows: Version 12.1.0.2.0 - Production
    Start Date                03-DEC-2015 12:27:51
    Uptime                    0 days 3 hr. 5 min. 21 sec
    Trace Level               off
    Security                  ON: Local OS Authentication
    SNMP                      OFF
    Listener Parameter File   C:\ORACLE\HOME\network\admin\listener.ora
    Listener Log File         C:\ORACLE\diag\tnslsnr\HOSTNAME\listener\alert\log.xml
    Listening Endpoints Summary...
      (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=HOSTNAME.domain)(PORT=1521)))
      (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(PIPENAME=\\.\pipe\EXTPROC1521ipc)))
      (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=HOSTNAME.domain)(PORT=2200))(Presentation=HTTP)(Session=RAW))
      (DESCRIPTION=(ADDRESS=(PROTOCOL=tcps)(HOST=HOSTNAME.domain)(PORT=5501))(Security=(my_wallet_directory=C:\ORACLE\admin\ecoomdb\xdb_wallet))(Presentation=HTTP)(Session=RAW))
      (DESCRIPTION=(ADDRESS=(PROTOCOL=tcps)(HOST=HOSTNAME.domain)(PORT=5500))(Security=(my_wallet_directory=C:\ORACLE\admin\ecoomdb\xdb_wallet))(Presentation=HTTP)(Session=RAW))
    Services Summary...
    Service "CLRExtProc" has 1 instance(s).
      Instance "CLRExtProc", status UNKNOWN, has 1 handler(s) for this service...
    Service "bibliodb.domain" has 1 instance(s).
      Instance "ecoomdb", status READY, has 1 handler(s) for this service...
    Service "ecoomdb.domain" has 1 instance(s).
      Instance "ecoomdb", status READY, has 1 handler(s) for this service...
    Service "ecoomdbXDB.domain" has 1 instance(s).
      Instance "ecoomdb", status READY, has 1 handler(s) for this service...
    Service "technodb.domain" has 1 instance(s).
      Instance "ecoomdb", status READY, has 1 handler(s) for this service...
    The command completed successfully

"lsnrctl services" output

    LSNRCTL for 64-bit Windows: Version 12.1.0.2.0 - Production on 03-DEC-2015 15:34:22

    Copyright (c) 1991, 2014, Oracle.  All rights reserved.

    Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=HOSTNAME.domain)(PORT=1521)))
    Services Summary...
    Service "CLRExtProc" has 1 instance(s).
      Instance "CLRExtProc", status UNKNOWN, has 1 handler(s) for this service...
        Handler(s):
          "DEDICATED" established:0 refused:0
             LOCAL SERVER
    Service "bibliodb.domain" has 1 instance(s).
      Instance "ecoomdb", status READY, has 1 handler(s) for this service...
        Handler(s):
          "DEDICATED" established:9 refused:0 state:ready
             LOCAL SERVER
    Service "ecoomdb.domain" has 1 instance(s).
      Instance "ecoomdb", status READY, has 1 handler(s) for this service...
        Handler(s):
          "DEDICATED" established:9 refused:0 state:ready
             LOCAL SERVER
    Service "ecoomdbXDB.domain" has 1 instance(s).
      Instance "ecoomdb", status READY, has 1 handler(s) for this service...
        Handler(s):
          "D000" established:8 refused:21 current:0 max:1022 state:ready
             DISPATCHER <machine: HOSTNAME, pid: 4040>
             (ADDRESS=(PROTOCOL=tcp)(HOST=HOSTNAME.domain)(PORT=49179))
    Service "technodb.domain" has 1 instance(s).
      Instance "ecoomdb", status READY, has 1 handler(s) for this service...
        Handler(s):
          "DEDICATED" established:9 refused:0 state:ready
             LOCAL SERVER
    The command completed successfully

listener.ora

    SID_LIST_LISTENER =
      (SID_LIST =
        (SID_DESC =
          (SID_NAME = CLRExtProc)
          (ORACLE_HOME = C:\ORACLE\HOME)
          (PROGRAM = extproc)
          (ENVS = "EXTPROC_DLLS=ONLY:C:\ORACLE\HOME\bin\oraclr12.dll")
        )
      )

    LISTENER =
      (DESCRIPTION_LIST =
        (DESCRIPTION =
          (ADDRESS = (PROTOCOL = TCP)(HOST = HOSTNAME.domain)(PORT = 1521))
          (ADDRESS = (PROTOCOL = IPC)(KEY = EXTPROC1521))
        )
      )

init.ora
(Trimmed, also contains other settings…)

db_domain=domain
db_name="ecoomdb"
dispatchers="(PROTOCOL=TCP) (SERVICE=ecoomdbXDB)"
local_listener=LISTENER*

netstat -a
(relevant selection)

 TCP    0.0.0.0:1521           ECONECOOMDB1:0         LISTENING
TCP    0.0.0.0:5500           ECONECOOMDB1:0         LISTENING
TCP    0.0.0.0:5501           ECONECOOMDB1:0         LISTENING
TCP    [::]:5500              ECONECOOMDB1:0         LISTENING
TCP    [::]:5501              ECONECOOMDB1:0         LISTENING

windows hosts file
(this file is empty)

Environment variables

  • PATH contains C:\ORACLE\HOME\bin
  • ORACLE_HOME = C:\ORACLE\HOME
  • ORACLE_SID = ecoomdb

Registry
HKEY_LOCAL_MACHINE\SOFTWARE\ORACLE\KEY_OraDB12Home1

  • ORACLE_HOME = C:\ORACLE\HOME
  • ORACLE_SID = ecoomdb

Related and duplicate (unanswered) questions/documents

Online resources

Stackexchange questions

Reddit threads

Oracle community

Best Answer

After contacting Oracle support, I've -kinda- been able to figure out what was going on. This server has 4 network adaptors, 1 adaptor connected to the network, and 3 unconfigured adaptors connected to an iSCSI storage. The problem seems to be caused by the web-client, or EM server (not sure here) trying to send the webpage request to the wrong adaptor.

For anyone facing the same issue, you can check this by browsing to the URL (using the hostname.domain) and then checking the listener.log file (-oracle_base-/diag/tnslsnr/-instance-/listener/trace/listener.log). For me, in the error message a different IP address than the servers main network adaptors IP address was shown. The IP address of one of the network adaptors connected to the iSCSI where shown. When omitting the IP addresses in my original post, I didn't spot this IP address difference.

18-DEC-2015 14:17:04 * http * (ADDRESS=(PROTOCOL=tcps)(HOST=<NOT SERVER IP address!>%14)(PORT=62119)) * handoff * http * 12518
TNS-12518: TNS:listener could not hand off client connection
 TNS-12560: TNS:protocol adapter error

When disabling the additional network adaptors, everything works fine.

Additionally, when going to "chrome://net-internals/#dns" in chrome, the IP addresses of all four network adaptors showed up for my hostname.

Apparently, this problem -does not- happen, when browsing to the webpage from another host. Also, nslookup returns only the correct IP address. So it seems this is not a DNS issue, but a local issue. I think this is not a full answer to the problem, as disabling network adaptors can't be a final solution, but I hope anyone facing this issue in the future has "more to go on" now...

Where does the discrepancy between IP address returned by the DNS server, and the IP addresses used by web browsers come from?

I've also made some additional changes that might have influenced the end-solution, so for completeness I'll include them. Not sure if relevant...:

  • Increased processes parameter of the database from 300 to 1000
  • Altered the "dispatchers" parameter by adding "(DISPATCHERS=5)"
Related Question