Once , i was trying to migrate OEM database from one database server to another.
1. We created the standby on the new server
2. During downtime, we synched it up
3. Opened the standby as primary
4. Tried re-pointing the connection string to new database using:
emctl config oms -store_repos_details -repos_conndesc "(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=ovpd-scan)(PORT=1521)))(LOAD_BALANCE=ON)(CONNECT_DATA=(SERVICE_NAME=CMPREP)))" -repos_user SYSMAN
This failed with the below error:
[oracle@ovp-v-oempd1 bin]$ emctl config oms -store_repos_details -repos_conndesc "(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=ovpd-scan)(PORT=1521)))(LOAD_BALANCE=ON)(CONNECT_DATA=(SERVICE_NAME=CMPREP)))" -repos_user SYSMAN
Oracle Enterprise Manager Cloud Control 12c Release 4
Copyright (c) 1996, 2014 Oracle Corporation. All rights reserved.
Error occurred. Check the log /u01/app/oracle/product/gc_inst1/em/EMGC_OMS1/sysman/log/secure.log
When I check in the secure.log it just says
secure.log
2016-01-18 14:31:17,589 [main] INFO util.EmctlUtil logp.251 - Connecting over t3s to: ovp-v-oempd1.compassminerals.com/7102 using id: weblogic
2016-01-18 14:32:23,368 [main] INFO util.EmctlUtil logp.251 - Unable to get mbean conn over t3s :null
2016-01-18 14:32:23,368 [main] INFO oms.StoreReposDetails logp.251 - Since there is a failure, rolling back the getLockConn
2016-01-18 14:32:23,373 [main] ERROR oms.StoreReposDetails logp.251 -
java.lang.NullPointerException
at weblogic.rjvm.ResponseImpl.unmarshalReturn(ResponseImpl.java:237)
at weblogic.rmi.internal.BasicRemoteRef.invoke(BasicRemoteRef.java:223)
at weblogic.management.remote.iiop.IIOPServerImpl_1036_WLStub.newClient(Unknown Source)
at weblogic.management.remote.common.RMIServerWrapper.newClient(ClientProviderBase.java:348)
at javax.management.remote.rmi.RMIConnector.getConnection(RMIConnector.java:2327)
at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:277)
at weblogic.management.remote.common.WLSRMIConnector.doConnect(WLSRMIConnector.java:152)
at weblogic.management.remote.common.WLSRMIConnector.access$100(WLSRMIConnector.java:29)
at oracle.sysman.emctl.util.EmctlUtil.getMBeanServerConn(EmctlUtil.java:775)
at oracle.sysman.emctl.config.oms.StoreReposDetails.processCmd(StoreReposDetails.java:248)
at oracle.sysman.emctl.config.oms.StoreReposDetails.main(StoreReposDetails.java:549)
Caused by: java.lang.NullPointerException
at oracle.security.jps.az.internal.runtime.policy.SystemPolicyImpl.(SystemPolicyImpl.java:85)
at oracle.security.jps.az.internal.runtime.service.PDPServiceImpl.initializeUncontrolledMode(PDPServiceImpl.java:570)
at oracle.security.jps.az.internal.runtime.service.PDPServiceImpl.initial(PDPServiceImpl.java:489)
at oracle.security.jps.az.internal.runtime.service.PDPServiceImpl.getSystemPolicy(PDPServiceImpl.java:888)
at oracle.security.jps.az.internal.runtime.s
I Initially thought its some issue with the weblogic, but I couldnot find any issues that I could relate to the above error.
I could connect fine using the same tns entry .
So .. Long story short, it turned out to be a DNS issue.
The DNS servers in our network are very crappy and unstable.
To resolve this issue I had to do the following:
2) Stopped the OMS using,
emctl stop oms -all -force
3) Killed remaining processes and restarted the oms
-> OMS services were started successfully
4) Temporarily changed the repository database IP address to old IP db address 10.177.10.10 and added an entry in /etc/hosts file
5) Forced a local hostname resolution on the OMS host by adding a line which points to old IP address of reposioty database 10.177.10.10 to new host newhost.oracle.com in /etc/hosts file.
6) Changed the host in the connection string using emctl on the OMS, from 10.177.10.10 to newhost.oracle.com
emctl config oms -store_repos_details -repos_conndesc '(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=newhost.oracle.com)(PORT=1521))(CONNECT_DATA=(SERVER=DEDICATED)(SERVICE_NAME=EMREP)))' -repos_user sysman
Which completed successfully
7) Stopped OMS
8) Changed the IP address of repository database back to new IP 10.177.10.50 and removed entries from /etc/hosts file (as DNS was configured to point to new IP address), also removed old IP address entries from /etc/hosts on OMS host.
9) Removed the line from the /etc/hosts of the OMS, allowing the use of the DNS to resolve newhost.oracle.com. Now newhost.oracle.com maps to 10.177.10.50.
10) Started OMS services successfully
This is a bit complex and very unpredictable, but helped us in this situation.
Hope this helps some one.
1. We created the standby on the new server
2. During downtime, we synched it up
3. Opened the standby as primary
4. Tried re-pointing the connection string to new database using:
emctl config oms -store_repos_details -repos_conndesc "(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=ovpd-scan)(PORT=1521)))(LOAD_BALANCE=ON)(CONNECT_DATA=(SERVICE_NAME=CMPREP)))" -repos_user SYSMAN
This failed with the below error:
[oracle@ovp-v-oempd1 bin]$ emctl config oms -store_repos_details -repos_conndesc "(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=ovpd-scan)(PORT=1521)))(LOAD_BALANCE=ON)(CONNECT_DATA=(SERVICE_NAME=CMPREP)))" -repos_user SYSMAN
Oracle Enterprise Manager Cloud Control 12c Release 4
Copyright (c) 1996, 2014 Oracle Corporation. All rights reserved.
Error occurred. Check the log /u01/app/oracle/product/gc_inst1/em/EMGC_OMS1/sysman/log/secure.log
When I check in the secure.log it just says
secure.log
2016-01-18 14:31:17,589 [main] INFO util.EmctlUtil logp.251 - Connecting over t3s to: ovp-v-oempd1.compassminerals.com/7102 using id: weblogic
2016-01-18 14:32:23,368 [main] INFO util.EmctlUtil logp.251 - Unable to get mbean conn over t3s :null
2016-01-18 14:32:23,368 [main] INFO oms.StoreReposDetails logp.251 - Since there is a failure, rolling back the getLockConn
2016-01-18 14:32:23,373 [main] ERROR oms.StoreReposDetails logp.251 -
java.lang.NullPointerException
at weblogic.rjvm.ResponseImpl.unmarshalReturn(ResponseImpl.java:237)
at weblogic.rmi.internal.BasicRemoteRef.invoke(BasicRemoteRef.java:223)
at weblogic.management.remote.iiop.IIOPServerImpl_1036_WLStub.newClient(Unknown Source)
at weblogic.management.remote.common.RMIServerWrapper.newClient(ClientProviderBase.java:348)
at javax.management.remote.rmi.RMIConnector.getConnection(RMIConnector.java:2327)
at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:277)
at weblogic.management.remote.common.WLSRMIConnector.doConnect(WLSRMIConnector.java:152)
at weblogic.management.remote.common.WLSRMIConnector.access$100(WLSRMIConnector.java:29)
at oracle.sysman.emctl.util.EmctlUtil.getMBeanServerConn(EmctlUtil.java:775)
at oracle.sysman.emctl.config.oms.StoreReposDetails.processCmd(StoreReposDetails.java:248)
at oracle.sysman.emctl.config.oms.StoreReposDetails.main(StoreReposDetails.java:549)
Caused by: java.lang.NullPointerException
at oracle.security.jps.az.internal.runtime.policy.SystemPolicyImpl.
at oracle.security.jps.az.internal.runtime.service.PDPServiceImpl.initializeUncontrolledMode(PDPServiceImpl.java:570)
at oracle.security.jps.az.internal.runtime.service.PDPServiceImpl.initial(PDPServiceImpl.java:489)
at oracle.security.jps.az.internal.runtime.service.PDPServiceImpl.getSystemPolicy(PDPServiceImpl.java:888)
at oracle.security.jps.az.internal.runtime.s
I Initially thought its some issue with the weblogic, but I couldnot find any issues that I could relate to the above error.
I could connect fine using the same tns entry .
So .. Long story short, it turned out to be a DNS issue.
The DNS servers in our network are very crappy and unstable.
To resolve this issue I had to do the following:
2) Stopped the OMS using,
emctl stop oms -all -force
3) Killed remaining processes and restarted the oms
-> OMS services were started successfully
4) Temporarily changed the repository database IP address to old IP db address 10.177.10.10 and added an entry in /etc/hosts file
5) Forced a local hostname resolution on the OMS host by adding a line which points to old IP address of reposioty database 10.177.10.10 to new host newhost.oracle.com in /etc/hosts file.
6) Changed the host in the connection string using emctl on the OMS, from 10.177.10.10 to newhost.oracle.com
emctl config oms -store_repos_details -repos_conndesc '(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=newhost.oracle.com)(PORT=1521))(CONNECT_DATA=(SERVER=DEDICATED)(SERVICE_NAME=EMREP)))' -repos_user sysman
Which completed successfully
7) Stopped OMS
8) Changed the IP address of repository database back to new IP 10.177.10.50 and removed entries from /etc/hosts file (as DNS was configured to point to new IP address), also removed old IP address entries from /etc/hosts on OMS host.
9) Removed the line from the /etc/hosts of the OMS, allowing the use of the DNS to resolve newhost.oracle.com. Now newhost.oracle.com maps to 10.177.10.50.
10) Started OMS services successfully
This is a bit complex and very unpredictable, but helped us in this situation.
Hope this helps some one.
Comments
Post a Comment