emctl config oms -store_repos_details command failed

Once , i was trying to migrate OEM database from one database server to another.

1. We created the standby on the new server
2. During downtime, we synched it up
3. Opened the standby as primary
4. Tried re-pointing the connection string to new database using:
emctl config oms -store_repos_details -repos_conndesc "(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=ovpd-scan)(PORT=1521)))(LOAD_BALANCE=ON)(CONNECT_DATA=(SERVICE_NAME=CMPREP)))" -repos_user SYSMAN 

This failed with the below error:

[oracle@ovp-v-oempd1 bin]$ emctl config oms -store_repos_details -repos_conndesc "(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=ovpd-scan)(PORT=1521)))(LOAD_BALANCE=ON)(CONNECT_DATA=(SERVICE_NAME=CMPREP)))" -repos_user SYSMAN 
Oracle Enterprise Manager Cloud Control 12c Release 4 
Copyright (c) 1996, 2014 Oracle Corporation. All rights reserved. 
Error occurred. Check the log /u01/app/oracle/product/gc_inst1/em/EMGC_OMS1/sysman/log/secure.log 

When I check in the secure.log it just says

secure.log 

2016-01-18 14:31:17,589 [main] INFO util.EmctlUtil logp.251 - Connecting over t3s to: ovp-v-oempd1.compassminerals.com/7102 using id: weblogic 
2016-01-18 14:32:23,368 [main] INFO util.EmctlUtil logp.251 - Unable to get mbean conn over t3s :null 
2016-01-18 14:32:23,368 [main] INFO oms.StoreReposDetails logp.251 - Since there is a failure, rolling back the getLockConn 
2016-01-18 14:32:23,373 [main] ERROR oms.StoreReposDetails logp.251 - 
java.lang.NullPointerException 
at weblogic.rjvm.ResponseImpl.unmarshalReturn(ResponseImpl.java:237) 
at weblogic.rmi.internal.BasicRemoteRef.invoke(BasicRemoteRef.java:223) 
at weblogic.management.remote.iiop.IIOPServerImpl_1036_WLStub.newClient(Unknown Source) 
at weblogic.management.remote.common.RMIServerWrapper.newClient(ClientProviderBase.java:348) 
at javax.management.remote.rmi.RMIConnector.getConnection(RMIConnector.java:2327) 
at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:277) 
at weblogic.management.remote.common.WLSRMIConnector.doConnect(WLSRMIConnector.java:152) 
at weblogic.management.remote.common.WLSRMIConnector.access$100(WLSRMIConnector.java:29) 

at oracle.sysman.emctl.util.EmctlUtil.getMBeanServerConn(EmctlUtil.java:775) 
at oracle.sysman.emctl.config.oms.StoreReposDetails.processCmd(StoreReposDetails.java:248) 
at oracle.sysman.emctl.config.oms.StoreReposDetails.main(StoreReposDetails.java:549) 
Caused by: java.lang.NullPointerException 
at oracle.security.jps.az.internal.runtime.policy.SystemPolicyImpl.(SystemPolicyImpl.java:85) 
at oracle.security.jps.az.internal.runtime.service.PDPServiceImpl.initializeUncontrolledMode(PDPServiceImpl.java:570) 
at oracle.security.jps.az.internal.runtime.service.PDPServiceImpl.initial(PDPServiceImpl.java:489) 
at oracle.security.jps.az.internal.runtime.service.PDPServiceImpl.getSystemPolicy(PDPServiceImpl.java:888) 
at oracle.security.jps.az.internal.runtime.s 


I Initially thought its some issue with the weblogic, but I couldnot find any issues that I could relate to the above error.

I could connect fine using the same tns entry .

So .. Long story short, it turned out to be a DNS issue.

The DNS servers in our network are very crappy and unstable.

To resolve this issue I had to do the following:


2) Stopped the OMS using, 

emctl stop oms -all -force 

3) Killed remaining processes and restarted the oms 
-> OMS services were started successfully 

4) Temporarily changed the repository database IP address to old IP db address 10.177.10.10 and added an entry in /etc/hosts file 

5) Forced a local hostname resolution on the OMS host by adding a line which points to old IP address of reposioty database 10.177.10.10 to new host newhost.oracle.com in /etc/hosts file. 

6) Changed the host in the connection string using emctl on the OMS, from 10.177.10.10 to newhost.oracle.com 

emctl config oms -store_repos_details -repos_conndesc '(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=newhost.oracle.com)(PORT=1521))(CONNECT_DATA=(SERVER=DEDICATED)(SERVICE_NAME=EMREP)))' -repos_user sysman 

Which completed successfully 

7) Stopped OMS 

8) Changed the IP address of repository database back to new IP 10.177.10.50 and removed entries from /etc/hosts file (as DNS was configured to point to new IP address), also removed old IP address entries from /etc/hosts on OMS host. 

9) Removed the line from the /etc/hosts of the OMS, allowing the use of the DNS to resolve newhost.oracle.com. Now newhost.oracle.com maps to 10.177.10.50. 

10) Started OMS services successfully 


This is a bit complex and very unpredictable, but helped us in this situation.

Hope this helps some one.