ORA-01017: invalid username/password; logon denied after upgrade 11.2

Over the last several weeks, I have been upgrading several database to 11.2 with and without broker.

I will write a step-by-step guide later and some of the problems I found.

The interesting thing about my last upgrade was an error (not surprise here, if all went smooth I will not have anything to write about) with the standby database.

The problem:

The error was quite self-explanatory:

ORACLE Instance ddddd - Archival Error. Archiver continuing.
Errors in file /opt/oracle/diag/rdbms/ssssss/dddddd/trace/ddddd_arc0_6420.trc:
ORA-01017: invalid username/password; logon denied
Error 1017 received logging on to the standby
------------------------------------------------------------
Check that the primary and standby are using a password file
and remote_login_passwordfile is set to SHARED or EXCLUSIVE,
and that the SYS password is same in the password files.
returning error ORA-16191
------------------------------------------------------------
Errors in file /opt/oracle/diag/rdbms/ssssss/dddddd/trace/ddddd_arc0_6420.trc:
ORA-16191: Primary log shipping client not logged on standby
FAL[server, ARC0]: Error 16191 creating remote archivelog file 'delta'
FAL[server, ARC0]: FAL archive failed, see trace file.
Errors in file /opt/oracle/diag/rdbms/ssssss/dddddd/trace/ddddd_arc0_6420.trc:
ORA-16055: FAL request rejected
ARCH: FAL archive failed. Archiver continuing

and the contents of /opt/oracle/diag/rdbms/ssssss/dddddd/trace/ddddd_arc0_6420.trc are:

ORA-01017: invalid username/password; logon denied
OCI_DBVER attribute value retrieval failed error=1017
*** 2010-06-26 08:51:58.413 4132 krsh.c
Error 1017 received logging on to the standby
------------------------------------------------------------
Check that the primary and standby are using a password file
and remote_login_passwordfile is set to SHARED or EXCLUSIVE,
and that the SYS password is same in the password files.
returning error ORA-16191
------------------------------------------------------------
*** 2010-06-26 08:51:58.413 869 krsu.c

The solution:

The first thing I did was to check the value of “remote_log_passwordfile” parameter. The value was correct (exclusive).

The second step was trying to connect as sys from one box the other and it was also fine.

What I did next was to recreate the orapwd files on primary and standby, but with two extra options:

orapwd file=orapwXXXXX password=YYYYYY entries=NN ignorecase=y force=y

With the ignorecase, we are telling oracle to revert to the previous behavior (all capitals). Force will allow to overwrite the current passwordfile if it exists.

After stopping and starting the primary and standby databases, all started to work as normal.

I think the problem was related to some capitals on the passwordfile when it was first created, but this is just a guess.

As always, comments are welcome.